How to Change Apigee Cassandra Passwords in Apigee Hybrid

[imesh]

Update - 4th Feb 2022:

• Apigee Cassandra credentials are also stored in Kubernetes secrets used by MART, runtime and synchronizer pods. Therefore, once Cassandra passwords are updated in the Cassandra database those components will not be able to communicate with Cassandra. 
• Therefore, this process would require a downtime.
• As of apigeectl v1.6.3, apigeectl does not support updating Cassandra passwords.
• Once Cassandra passwords are updated in Cassandra database, we need to manually update Cassandra passwords in secrets used by MART, runtime and synchronizer pods.

In Apigee Hybrid the Cassandra credentials are defined in the overrides.yaml file or their default values will be taken from the config/values.yaml file:

cassandra:
  ...
  auth:
    default:
      password: "iloveapis123"
    admin:
      password: "iloveapis123"
    ddl:
      password: "iloveapis123"
    dml:
      password: "iloveapis123"
    jmx:
      username: "jmxuser"
      password: "iloveapis123"
    jolokia:
      username: "jolokiauser"
      password: "iloveapis123"

Once the installation is complete, apigeectl does not allow changing them. You could find this information on the Changing the default passwords in the overrides file documentation page:

Note: You can only change these defaults at the time of initial setup. Password rotation or change after hybrid setup is not possible.

Background

• To update Cassandra passwords in an Apigee hybrid environment, we need to update them in the Cassandra database and in Kubernetes secrets used by Apigee MART, runtime and Synchronizer pods.
• Once Cassandra passwords are updated in the Cassandra database, Apigee MART, runtime and Synchronizer will not be able to connect to Cassandra until the passwords are updated in the relevant Kubernetes secrets and pods are restarted.
• Therefore, this process would require a downtime.
• Currently, apigeectl does not provide a command or a set of commands for executing above steps.
• That's the reason why Apigee documentation has mentioned above.

How to Manually Update Cassandra Passwords

In Cassandra, user credentials are stored in the database itself. Hence, once Cassandra is installed we need to connect to the Cassandra database and change the passwords of those users. This can be done in following steps:

1. Start a cqlsh client pod:

kubectl run -i --tty --restart=Never --rm --image google/apigee-hybrid-cassandra-client:1.0.0 cqlsh

2. Connect to the cqlsh shell using ddl_user:

cqlsh apigee-cassandra-default-0.apigee-cassandra-default.apigee.svc.cluster.local -u ddl_user --ssl

3. List all users:

ddl_user@cqlsh> SELECT * FROM system_auth.roles;

An example output:

 role       | can_login | is_superuser | member_of | salted_hash
------------+-----------+--------------+-----------+--------------------------------------------------------------
  cassandra |      True |         True |      null | $2a$10$98HgK1FA97DaJO7apBU3uee7VzAQvmYwuCtNwhkDuNMl6vrxnHsE2
    jmxuser |      True |        False |      null | $2a$10$SsHV1ezshryrHPUJvwV.auoJhMYToSPyoLNZbZWE4fR8ghNaVaHvq
   dml_user |      True |        False |      null | $2a$10$HVFy9eg6CaZbAOwGZT5iOeZuMtXzuaVqICDsyTSntiaeiLC1IS0sm
 admin_user |      True |         True |      null | $2a$10$z/tGMp8QvyQHwHIbyfXLo.pF0hWefRaQ.v8v3SRNrX4377uvOR6P.
   ddl_user |      True |        False |      null | $2a$10$LabVlsAr7Cm2IawaJbIcweBqyOdqf0Dm6XyjUAmbCPoiIJ9uQux.u

4. Now, update the password of a specific user by executing below SQL, set username and password accoringly:

ALTER USER ddl_user WITH PASSWORD 'new-password';

5. Exit from cqlsh:

quit

6. Verify the new password given above by connecting to the cqlsh again:

cqlsh apigee-cassandra-default-0.apigee-cassandra-default.apigee.svc.cluster.local -u ddl_user --ssl