Baas/Cassandra creating large log files

[zubermohammad]

Hello Geeks,

Recently we came across a scenario where Baas and Cassandra create really huge log files. Huge as in 20 GB file for single day log. We have OPDK installation where the external public doesn't access to any APIs.

This is the file that was taking up complete space allotted to /opt.

/opt/apigee/var/log/apigee-tomcat/catalina.out

This is causing heap overflow in Java and killing the process/deamon on linux.

Any idea why these huge files are being generated. Is it possible that a DOS attack was triggered? If so how can we handle such scenarios and protect against attacks?

Any help is appreciated.

[bkrishnankutty]

Can you do a long listing of Cassandra logs and tomcat logs ?

there should be auto log rotate, which should address the log file size.

[#3]

What was being logged in the file? The catalina.out file is Tomcat's main application log file and BaaS runs inside of Tomcat, so all of the BaaS application logs are here. It's hard to give you an answer to why, without seeing what was actually being logged.

[zubermohammad]

Hi @Michael Russo,

This is a part of the "catalina.out" file. This file was 20 GB in size. I broke it down into parts to download.

When analysed the first part I got a lot of repeated entries for exception:

2016-11-20 05:40:21,698 [pool-484-thread-1] ERROR org.apache.usergrid.services.notifications.apns.RejectedAPNsListener- Failed to register push connection
java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:101)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:193)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:200)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1029)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:496)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:481)
	at io.netty.handler.ssl.SslHandler.connect(SslHandler.java:386)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:496)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:481)
	at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:496)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:481)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:463)
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:849)
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:199)
	at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:165)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)

After searching for this exception I landed up on a bug that was reported in Usergrid, captured in support ticket on link here.

https://apigeesc.atlassian.net/browse/APIBAAS-1525

I don't have access to this portal, so can someone having access please analyze the bug above and tell me how we can tackle this situation?

Attaching the logs for your reference. Again this is just a 5 MB file. Still need to go through the end of 20 GB :(.catalinaout.txt

[#5]

Which version of OPDK are you running?

[#6]

I can also see from your logs that Cassandra is unavailable ( either network or your Cassandra servers are down). It appears the cause of your problem with the log file growing is due to other contributing factors that first need resolved.

[zubermohammad]

@Michael Russo,

We have 4.16.05 version installed. After some troubleshooting, I found that the DNS was not configured on casssandra servers.

When I added the host entry for gateway.sandbox.push.apple.com and triggered some notifications to APNS the logs didn't grow as they were doing earlier.

But still can't figure out why we were getting such huge log entries for same exception.

[zubermohammad]

This is the bug that was reported in Usergrid community, regarding the same exception.

https://issues.apache.org/jira/browse/USERGRID-529