INFO JVM Arguments . . . -XX:+HeapDumpOnOutOfMemoryError - is the node out of Memory??

[pradeepshanmugh]

Hi,

Does the below INFO message that appears on server logs indicate that the node has run out of memory ?

INFO 15:08:37 JVM Arguments: [-Dorg.xerial.snappy.tempdir=/app/apigee/data/apigee-cassandra/data/tmp, -Dorg.xerial.snappy.tempdir=/app/apigee/data/apigee-cassandra/data/tmp, -ea, -javaagent:/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms8192M, -Xmx8192M, -Xmn1200M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:CompileCommandFile=/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../conf/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -Djava.net.preferIPv4Stack=true, -Dcom.sun.management.jmxremote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/app/apigee/data/apigee-cassandra/jmxremote.password, -javaagent:/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../lib/jolokia-jvm-1.3.5-agent.jar=host=0.0.0.0, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../logs, -Dcassandra.storagedir=/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../data, -Dcassandra-foreground=yes]

This entry is being used by theOps team to suggest that the node has run out of memory.

My belief is this is just an info message by a service, stating that the JVM arguments provided to it instructs it to create a dump of the heap if the service ran out of memory and that the appearance of this message itself does not indicate that the node has run out of memory.

Does that INFO entry mean I am running out memory?

[#2]

no, this is heap memory issue. You can increase your heap memory, if already not done. Else restart of the component will fix that issue.

for detail step to increase the memory can refer to

https://docs.apigee.com/private-cloud/v4.19.01/modifying-java-settings

[pradeepshanmugh]

Can you please clarify what makes you say so?

doing more research :
According to documentation

https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html

it states that this is just a command line option

[#4]

the error shows heap memory issue, and the information in "Xmx8192M, -Xmn1200M" are the configuration information of heap memory. We earlier faced heap memory issues, for which we did restart for immediate resolution and for long term we have increased the heap memory.

[pradeepshanmugh]

I am still not seeing the connection unfortunately.

I think because the JVM argument has the word error in it - its easily being confused, that this as an error message.

From all the Oracle JVM documentation online, if there was a heap memory issue and if the java service was started with the command line option to capture a dump of the heap, then the error log would have an entry that would resemble

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid2262.hprof ...
Heap dump file created [531535128 bytes in 14.691 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
        at ConsumeHeap.main(ConsumeHeap.java:32)

See B.1.2 : https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html

Depending on the config - a dump file would be created at a location that would resemble

java_pidXXXXX.hprof 

and present a stack trace.

I am not suggesting that either a) there is NO heap memory issue / b) that there is one.

What I am seeking to clarify is what that INFO entry actually means.

Note too that the entry just doe not talk about Heap Memory.
It also talks about

• Thread and Thread Priorities
• String Table Size
• A lot of entries regarding CMS***

Is there any documentation online around runtime error messaging as seen on the apigee node logs that help identify what entries to look for for memory issues?