Apigee Edge OPDK : How to remove cassandra node without affecting traffic?

How to decommission Cassandra node in Apigee Edge OPDK without causing live traffic disruption ? Anything to keep in mind before decommissioning cassandra nodes in Apigee Edge OPDK setup ?

~~Q:S:TC~~

Solved Solved
0 2 1,685
1 ACCEPTED SOLUTION

Not applicable

Below is the procedure. Please notice that the steps below could vary based on a number of factors such as topology design, existing Cassandra cluster configuration, API volume at the time of executing the procedure, among others.

If you require assistance with this procedure, please reach out to Google Technical Solutions (aka Apigee Customer Success).

The example below shows how to remove 3 nodes. It assumes you are scaling down the ring in the increment of the replication factor (3):

1. Remove Cassandra references from Pods

1.1. Update your existing response file to exclude nodes to be removed.

Current response file:

IP1=<node 1>
IP2=<node 2>
IP3=<node 3>
IP10=<node 10>
IP11=<node 11>
IP12=<node 12>
HOSTIP="$(hostname -i)"
...
CASS_HOSTS="$IP1:1,1 $IP10:1,1 $IP2:1,1 $IP11:1,1 $IP3:1,1 $IP12:1,1"
...
BIND_ON_ALL_INTERFACES="y"

Updated response file:

IP1=<node 1>
IP2=<node 2>
IP3=<node 3>
HOSTIP="$(hostname -i)"
...
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1"
...
BIND_ON_ALL_INTERFACES="y"

1.2. Update configuration of existing Management Server nodes:

/opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile

2. Restart components

Perform a rolling restart of all Edge components (R, MP, QS, PS):

/opt/apigee/apigee-service/bin/apigee-all restart 

The restart will force components to read latest Pod wiring information and remove references to existing Cassandra nodes.

3. Decommission Cassandra nodes

For each Cassandra node to remove, one node at the time, execute decommission command:

/opt/apigee/apigee-cassandra/bin/nodetool <options> decommission

Options:
( -h | --host ) <host name> | <ip address>
( -p | --port ) <port number>
( -pw | --password ) <password >
( -u | --username ) <user name>

Decommission command deactivates a node by streaming its data to another node.

4. Update local configuration on remaining Cassandra nodes

4.1. Update configuration of remaining Cassandra nodes:

/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile

5. Uninstall Cassandra

If required, uninstall Cassandra from nodes removed from the ring. Use the procedure described on the official documentation:

http://docs.apigee.com/private-cloud/latest/uninstalling-edge

View solution in original post

2 REPLIES 2

Not applicable

Below is the procedure. Please notice that the steps below could vary based on a number of factors such as topology design, existing Cassandra cluster configuration, API volume at the time of executing the procedure, among others.

If you require assistance with this procedure, please reach out to Google Technical Solutions (aka Apigee Customer Success).

The example below shows how to remove 3 nodes. It assumes you are scaling down the ring in the increment of the replication factor (3):

1. Remove Cassandra references from Pods

1.1. Update your existing response file to exclude nodes to be removed.

Current response file:

IP1=<node 1>
IP2=<node 2>
IP3=<node 3>
IP10=<node 10>
IP11=<node 11>
IP12=<node 12>
HOSTIP="$(hostname -i)"
...
CASS_HOSTS="$IP1:1,1 $IP10:1,1 $IP2:1,1 $IP11:1,1 $IP3:1,1 $IP12:1,1"
...
BIND_ON_ALL_INTERFACES="y"

Updated response file:

IP1=<node 1>
IP2=<node 2>
IP3=<node 3>
HOSTIP="$(hostname -i)"
...
CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1"
...
BIND_ON_ALL_INTERFACES="y"

1.2. Update configuration of existing Management Server nodes:

/opt/apigee/apigee-setup/bin/setup.sh -p ms -f updatedConfigFile

2. Restart components

Perform a rolling restart of all Edge components (R, MP, QS, PS):

/opt/apigee/apigee-service/bin/apigee-all restart 

The restart will force components to read latest Pod wiring information and remove references to existing Cassandra nodes.

3. Decommission Cassandra nodes

For each Cassandra node to remove, one node at the time, execute decommission command:

/opt/apigee/apigee-cassandra/bin/nodetool <options> decommission

Options:
( -h | --host ) <host name> | <ip address>
( -p | --port ) <port number>
( -pw | --password ) <password >
( -u | --username ) <user name>

Decommission command deactivates a node by streaming its data to another node.

4. Update local configuration on remaining Cassandra nodes

4.1. Update configuration of remaining Cassandra nodes:

/opt/apigee/apigee-setup/bin/setup.sh -p c -f updatedConfigFile

5. Uninstall Cassandra

If required, uninstall Cassandra from nodes removed from the ring. Use the procedure described on the official documentation:

http://docs.apigee.com/private-cloud/latest/uninstalling-edge

Great answer, Thank you @Maudrit , + 1