This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Log in to ask a question

JMX authentication error for cassandra

Posted on 08-23-2018 10:55 AM
ApigeeUser777
Bronze 3
Bronze 3
Reply posted on --/--/---- --:-- AM

Hi, I am facing the following error after enabling the JMX authentication for cassandra component as mentioned in the apigee docs " https://docs.apigee.com/private-cloud/v4.18.05/how-monitor "

Did anyone face this issue and able to solve it? We are using sitescope for monitoring.......

Checking the status of Cassandra before shutdown Disabling thrift... error: Authentication failed! Credentials required -- StackTrace -- java.lang.SecurityException: Authentication failed! Credentials required at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211) at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163) at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227) at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232) at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199) at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy7.newClient(Unknown Source) at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192) at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:159) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:350) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206) Unable to disable thrift. Disabling binary... error: Authentication failed! Credentials required -- StackTrace -- java.lang.SecurityException: Authentication failed! Credentials required at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211) at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163) at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227) at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232) at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199) at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy7.newClient(Unknown Source) at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192) at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:159) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:350) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206) Unable to disable binary. Disabling gossip... error: Authentication failed! Credentials required -- StackTrace -- java.lang.SecurityException: Authentication failed! Credentials required at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211) at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163) at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227) at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232) at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199) at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy7.newClient(Unknown Source) at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192) at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:159) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:350) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206) Unable to disable gossip. Draining... error: Authentication failed! Credentials required -- StackTrace -- java.lang.SecurityException: Authentication failed! Credentials required at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211) at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163) at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227) at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232) at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199) at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) at com.sun.proxy.$Proxy7.newClient(Unknown Source) at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192) at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:159) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:350) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206) Could not drain.done

0 8 2,006
Topic Labels
0 Likes
8 REPLIES 8

Posted on --/--/---- --:-- AM
mrshubhammahaja
New Member
Reply posted on --/--/---- --:-- AM

Hi Hemanth,

Are you facing the same problem ?

https://community.apigee.com/questions/59811/backup-not-working-after-enabling-cassandra-jmx-au.html

0 Likes

Posted on --/--/---- --:-- AM
ApigeeUser777
Bronze 3
Bronze 3
Reply posted on --/--/---- --:-- AM

Yes @shubham Mahajan Iam facing the same issue, The stop and backup action broke, I tried all the fixes you have mentioned in the link above, I am still facing the same issue.

I enabled the RMI settings in management.properties file and restarted the component, but still faced the same issue.

# ################ RMI User authentication ################ # com.sun.management.jmxremote.authenticate=true

# ################ RMI Password file location ################## # com.sun.management.jmxremote.password.file=$APIGEEroot/data/apigee-cassandra/jmxremote.password # Specifies location for password file

# ################ RMI Access file location ##################### # com.sun.management.jmxremote.access.file=$APIGEEroot/data/apigee-cassandra/jmxremote.access

I am I missing something? or are there any other fixes I should try ?? we are currently using version 4.18.01

0 Likes

Posted on --/--/---- --:-- AM
mrshubhammahaja
New Member
Reply posted on --/--/---- --:-- AM

Found a solution in last after too much of R&D.

1. Follow Documentation in https://docs.apigee.com/private-cloud/v4.17.05/how-monitor#apachecassandra to enable JMX authentication.
2. Change the location of filepath to new created files (jmxremote.access and jmxremote.password) in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64/jre/lib/management/management.properties ( or specific java version) so that it will take those files as input file because $LOCAL_JMX is 'yes' in cassandra-env.sh file.
3. Change the backup file in /opt/apigee/apigee-cassandra-2.1.16-0.0.1077/lib/actions and allow nodetool to pass the credential -u cassandra -pw password. Modified backup file is attached below.
4.Restart the server and you will be able to backup.

As per my understanding issue is due to nodetool, as nodetool needs authentication as an argument input but apigee backup file is not taking credential as input at all.

https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureNodetool.html (Cassandra Documentation - same for older version).

So I changed the backup file accordingly, you can find the attached file for the reference.

As Upload is not allowing .sh so changing the extension to .txt.

backup-original.txt - Original Backup File

backup-solution.txt - Changed Backup File.

backup-original.txt (1.6 kB) backup-solution.txt

(1.6 kB)

0 Likes

Posted on --/--/---- --:-- AM
mrshubhammahaja
New Member
Reply posted on --/--/---- --:-- AM

Can you share the steps you have followed ?

0 Likes

Posted on --/--/---- --:-- AM
ApigeeUser777
Bronze 3
Bronze 3
In response to mrshubhammahaja
Reply posted on --/--/---- --:-- AM

Sorry for the late response, We raised a ticket with google they are looking in to it and asked us to turn off JMX authentication for Cassandra for now. Thanks for your help @Shubham Mahajan, will update the link if they find any solution.

0 Likes

Posted on --/--/---- --:-- AM
michaelmclean
New Member
In response to ApigeeUser777
Reply posted on --/--/---- --:-- AM

@hemanth did you get a reply? what was the solution?

0 Likes

Posted on --/--/---- --:-- AM
mrshubhammahaja
New Member
Reply posted on --/--/---- --:-- AM

@Michael McLean They have updated the solution in the newer version with the workaround. But you can try the above solution and let me know if it didn't work for you because I am using the same solution for myself.

0 Likes

Posted on --/--/---- --:-- AM
Former Community Member
Not applicable
Reply posted on --/--/---- --:-- AM

Hi Team, we just ran into this problem trying to upgrade from 4.19.01 to 4.19.06 and this doc was very helpful in getting to an answer...

We had noticed Cassandra would fail draining every time we restarted the services “Could not drain…” and during the backups that occur during an upgrade would get invalid user/password

In the file /opt/apigee/apigee-cassandra/lib/actions/backup we decided to echo out the values for JMX_USERNAME and JMX_PASSWORD and when running they were showing us username was “FOO” and password were “BAR”... Upon searching for these values, we found them to exist in the file /opt/apigee/apigee-cassandra/lib/settings.

Once we changed the username and password values to our JMX values in /opt/apigee/apigee-cassandra/lib/settings , backup and restart worked as expected and would not throw errors stopping starting the services or running backup… Hope this helps the next soul fighting the issues!

0 Likes