Backup not working after enabling Cassandra JMX authentication .

[mrshubhammahaja]

After enabling the Cassandra JMX remote Authentication, backup is not working.

/opt/apigee/apigee-service/bin/apigee-service apigee-cassandra backup

Below error is coming after we enable the JMX authentication

error: Authentication failed! Credentials required
-- StackTrace --
java.lang.SecurityException: Authentication failed! Credentials required
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:211)
at com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:163)
at sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:227)
at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
at javax.management.remote.rmi.RMIJRMPServerImpl$ExportedWrapper.newClient(RMIJRMPServerImpl.java:253)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:835)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy7.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:192)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:159)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:350)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)


Document Refereces followed:-
For Authentication Enabling :- https://docs.apigee.com/private-cloud/v4.18.01/how-monitor

For Backup:- https://docs.apigee.com/private-cloud/v4.18.01/how-perform-backup


APIGEE Version - 14.17.05

[mrshubhammahaja]

Found a solution in last after too much of R&D.

1. Follow Documentation in https://docs.apigee.com/private-cloud/v4.17.05/how-monitor#apachecassandra to enable JMX authentication.
2. Change the location of filepath to new created files (jmxremote.access and jmxremote.password) in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64/jre/lib/management/management.properties ( or specific java version) so that it will take those files as input file because $LOCAL_JMX is 'yes' in cassandra-env.sh file.
3. Change the backup file in /opt/apigee/apigee-cassandra-2.1.16-0.0.1077/lib/actions and allow nodetool to pass the credential -u cassandra -pw password. Modified backup file is attached below.
4.Restart the server and you will be able to backup.

As per my understanding issue is due to nodetool, as nodetool needs authentication as an argument input but apigee backup file is not taking credential as input at all.

https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureNodetool.html (Cassandra Documentation - same for older version).

So I changed the backup file accordingly, you can find the attached file for the reference.

As Upload is not allowing .sh so changing the extension to .txt.

backup-original.txt - Original Backup File

backup-solution.txt - Changed Backup File.

[neeladrisomadde]

Hi Shubham,

I was referring to the following doc:

https://docs.apigee.com/private-cloud/v4.19.01/enable-cassandra-authentication#enablecassandraauthen...

Can you please clarify my following querries?:

1. Is there any inter-dependencies between the Cassandra authentication and JMX authentication in Cassandra?

2. Can you please elaborate on the impact on the traffic while we are enabling the JMX authentication in Cassandra servers?

3. If we upscale Cassandra later, would the changes get automatically reflected in the new nodes? Or do we have to manually enable it again?

Thanks in advance,

Neeladri