How to decommission Cassandra node in Apigee Edge OPDK without causing live traffic disruption ? Anything to keep in mind before decommissioning cassandra nodes in Apigee Edge OPDK setup ?
~~Q:S:TC~~
Solved! Go to Solution.
Below is the procedure. Please notice that the steps below could vary based on a number of factors such as topology design, existing Cassandra cluster configuration, API volume at the time of executing the procedure, among others.
If you require assistance with this procedure, please reach out to Google Technical Solutions (aka Apigee Customer Success).
The example below shows how to remove 3 nodes. It assumes you are scaling down the ring in the increment of the replication factor (3):
1. Remove Cassandra references from Pods
1.1. Update your existing response file to exclude nodes to be removed.
Current response file:
IP1=<node 1> IP2=<node 2> IP3=<node 3> IP10=<node 10> IP11=<node 11> IP12=<node 12> HOSTIP="$(hostname -i)" ... CASS_HOSTS="$IP1:1,1 $IP10:1,1 $IP2:1,1 $IP11:1,1 $IP3:1,1 $IP12:1,1" ... BIND_ON_ALL_INTERFACES="y"
Updated response file:
IP1=<node 1> IP2=<node 2> IP3=<node 3> HOSTIP="$(hostname -i)" ... CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" ... BIND_ON_ALL_INTERFACES="y"
1.2. Update configuration of existing Management Server nodes:
/opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile
2. Restart components
Perform a rolling restart of all Edge components (R, MP, QS, PS):
/opt/apigee/apigee-service/bin/apigee-all restart
The restart will force components to read latest Pod wiring information and remove references to existing Cassandra nodes.
3. Decommission Cassandra nodes
For each Cassandra node to remove, one node at the time, execute decommission command:
/opt/apigee/apigee-cassandra/bin/nodetool <options> decommission Options: ( -h | --host ) <host name> | <ip address> ( -p | --port ) <port number> ( -pw | --password ) <password > ( -u | --username ) <user name>
Decommission command deactivates a node by streaming its data to another node.
4. Update local configuration on remaining Cassandra nodes
4.1. Update configuration of remaining Cassandra nodes:
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
5. Uninstall Cassandra
If required, uninstall Cassandra from nodes removed from the ring. Use the procedure described on the official documentation:
http://docs.apigee.com/private-cloud/latest/uninstalling-edge
Below is the procedure. Please notice that the steps below could vary based on a number of factors such as topology design, existing Cassandra cluster configuration, API volume at the time of executing the procedure, among others.
If you require assistance with this procedure, please reach out to Google Technical Solutions (aka Apigee Customer Success).
The example below shows how to remove 3 nodes. It assumes you are scaling down the ring in the increment of the replication factor (3):
1. Remove Cassandra references from Pods
1.1. Update your existing response file to exclude nodes to be removed.
Current response file:
IP1=<node 1> IP2=<node 2> IP3=<node 3> IP10=<node 10> IP11=<node 11> IP12=<node 12> HOSTIP="$(hostname -i)" ... CASS_HOSTS="$IP1:1,1 $IP10:1,1 $IP2:1,1 $IP11:1,1 $IP3:1,1 $IP12:1,1" ... BIND_ON_ALL_INTERFACES="y"
Updated response file:
IP1=<node 1> IP2=<node 2> IP3=<node 3> HOSTIP="$(hostname -i)" ... CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1" ... BIND_ON_ALL_INTERFACES="y"
1.2. Update configuration of existing Management Server nodes:
/opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile
2. Restart components
Perform a rolling restart of all Edge components (R, MP, QS, PS):
/opt/apigee/apigee-service/bin/apigee-all restart
The restart will force components to read latest Pod wiring information and remove references to existing Cassandra nodes.
3. Decommission Cassandra nodes
For each Cassandra node to remove, one node at the time, execute decommission command:
/opt/apigee/apigee-cassandra/bin/nodetool <options> decommission Options: ( -h | --host ) <host name> | <ip address> ( -p | --port ) <port number> ( -pw | --password ) <password > ( -u | --username ) <user name>
Decommission command deactivates a node by streaming its data to another node.
4. Update local configuration on remaining Cassandra nodes
4.1. Update configuration of remaining Cassandra nodes:
/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile
5. Uninstall Cassandra
If required, uninstall Cassandra from nodes removed from the ring. Use the procedure described on the official documentation:
http://docs.apigee.com/private-cloud/latest/uninstalling-edge
Great answer, Thank you @Maudrit , + 1