Hi,
Does the below INFO message that appears on server logs indicate that the node has run out of memory ?
INFO 15:08:37 JVM Arguments: [-Dorg.xerial.snappy.tempdir=/app/apigee/data/apigee-cassandra/data/tmp, -Dorg.xerial.snappy.tempdir=/app/apigee/data/apigee-cassandra/data/tmp, -ea, -javaagent:/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms8192M, -Xmx8192M, -Xmn1200M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:CompileCommandFile=/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../conf/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -Djava.net.preferIPv4Stack=true, -Dcom.sun.management.jmxremote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/app/apigee/data/apigee-cassandra/jmxremote.password, -javaagent:/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../lib/jolokia-jvm-1.3.5-agent.jar=host=0.0.0.0, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../logs, -Dcassandra.storagedir=/app/apigee/apigee-cassandra-2.1.16-0.0.1123/bin/../data, -Dcassandra-foreground=yes]
This entry is being used by theOps team to suggest that the node has run out of memory.
My belief is this is just an info message by a service, stating that the JVM arguments provided to it instructs it to create a dump of the heap if the service ran out of memory and that the appearance of this message itself does not indicate that the node has run out of memory.
Does that INFO entry mean I am running out memory?
no, this is heap memory issue. You can increase your heap memory, if already not done. Else restart of the component will fix that issue.
for detail step to increase the memory can refer to
https://docs.apigee.com/private-cloud/v4.19.01/modifying-java-settings
Can you please clarify what makes you say so?
doing more research :
According to documentation
https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html
it states that this is just a command line option
the error shows heap memory issue, and the information in "Xmx8192M, -Xmn1200M" are the configuration information of heap memory. We earlier faced heap memory issues, for which we did restart for immediate resolution and for long term we have increased the heap memory.
I am still not seeing the connection unfortunately.
I think because the JVM argument has the word error in it - its easily being confused, that this as an error message.
From all the Oracle JVM documentation online, if there was a heap memory issue and if the java service was started with the command line option to capture a dump of the heap, then the error log would have an entry that would resemble
java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid2262.hprof ... Heap dump file created [531535128 bytes in 14.691 secs] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at ConsumeHeap$BigObject.(ConsumeHeap.java:22) at ConsumeHeap.main(ConsumeHeap.java:32)
See B.1.2 : https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html
Depending on the config - a dump file would be created at a location that would resemble
java_pidXXXXX.hprof
and present a stack trace.
I am not suggesting that either a) there is NO heap memory issue / b) that there is one.
What I am seeking to clarify is what that INFO entry actually means.
Note too that the entry just doe not talk about Heap Memory.
It also talks about
Is there any documentation online around runtime error messaging as seen on the apigee node logs that help identify what entries to look for for memory issues?