Cassandra node not added to ring

Hello Team,

I have a Apigee AIO private edge setup. Now I want to add another Cassandra node to this setup. I followed the step mentioned in https://docs.apigee.com/private-cloud/v4.19.01/adding-cassandra-nodes.

The cassandra nodes were started, but in the cassandra ring (using nodetool ring command) I can only see one node, which is the new node which I added. I don't see the previously setup cassandra node.

Also the management server is not able to access the cassandra datastore and gives an cassandra.thrift.UnavailableException

2021-04-14 08:39:00,654 org:myorg env: target:/v1/organizations/myorg/developers contextId: action: qtp1495161082-79 ERROR c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.trackError() : com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=None(0.0.0.0):0, latency=1(1), attempts=1]UnavailableException() com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=None(0.0.0.0):0, latency=1(1), attempts=1]UnavailableException()

org.apache.cassandra.thrift.UnavailableException: null

All the values in the configuration files contains the IP address of both nodes as per the guidelines mentioned in the above link.

Please let me know if anyone has faced this issue and also any help on this regard will be useful.

apigee version: 4.50.00

cassandra version: 2.1.22

Regards,

Mukunda

1 4 887
4 REPLIES 4

Not applicable

I would suggest not to try with aio. Cassandra ring should be 3X.

We observed the very same error message in a 51 node installation as well, having 27 CS nodes globally distributed across 9 data centers (5 locations / network zone, 4 of them having DR location, with 3 CS nodes in each location / network zone).

Hence 1-node installation should not be the root cause.

Any update, as we observe it still in Apigee 4.51 (without patch yet)?
Recommended way to resolve, and any option to auto-heal?
It's a huge risk for us having some hundred nodes in various stages and datacenters distributed globally...

Dear all,

 

we observed a similar error in our 9-node-installation: when we installed cassandra on node 1, it was part of the ring. When we then installed cassandra on node 2, only the second cassandra was part of the ring and the first cassandra was kicked out.

 

We checked the ring status with nodetool and saw that the token (a kind if "id") of the cassandra that was part of the ring, was always set to 0. 

So the behavior was basically:

- Cassandra 1 was part of the ring with token 0

- Cassandra 2 also obtained token 0. As a consequence, cassandra 1 was removed from the ring and cassandra 2 took its place for token 0.

 

For now, we fixed this problem by manually changing the tokens of the single cassandras with "nodetool move".

 

We do not know, though, was causes the install process to always get back to the same token. It would be great if you could find out.

 

Best,

Patrick