Adding new cassandra nodes results in failure

[aruljothi]

Trying to add 3 new cassandra nodes to a datacenter which already has 3 cassandra nodes

CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1 $IP4:1,2 $IP5:1,2 $IP6:1,2"

I am adding "$IP4:1,2 $IP5:1,2 $IP6:1,2" nodes

I found that $IP4L1,2 is added and setup properly, where as "$IP5:1,2 $IP6:1,2" has following error.

ERROR 12:51:40 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Bootstrapping to existing token 0 is not allowed (decommission/removenode the old node first).
at org.apache.cassandra.dht.BootStrapper.getBootstrapTokens(BootStrapper.java:108) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:895) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.16.jar:2.1.16]
Bootstrapping to existing token 0 is not allowed (decommission/removenode the old node first).
Fatal configuration error; unable to start server. See log for stacktrace.

[aruljothi · Accepted solution]

This is due to the way CASS_HOSTS config is updated.

It is clearly mentioned in https://docs.apigee.com/private-cloud/v4.18.01/adding-cassandra-nodes that "Important: Add each new Cassandra node to CASS_HOSTS after an existing node."

As per above statement, only $IP4L1,2 is added after an existing node $IP3:1,1 where as $IP5:1,2 $IP6:1,2 is not after existing node and hence the failure. please change the CASS_HOSTS config as shown below to resolve the issue.

CASS_HOSTS="$IP1:1,1 $IP4:1,2 $IP2:1,1 $IP5:1,2 $IP3:1,1 $IP6:1,2"

[pmallikarjuna]

HI Arujothi, i am facing exactly same issue while adding a new Cassandra node. i tried same format as you specified below but it could not resolved my issue. i'm using 4.19.01 version apigee and it is failing while adding 3rd node( IP6). Could you please let me know if you have any other solution to get rid of this issue?

CASS_HOSTS="$IP1:1,1 $IP4:1,2 $IP2:1,1 $IP5:1,2 $IP3:1,1 $IP6:1,2"

[pmallikarjuna]

Never mind. I have figured out the issue, due to firewall rules the communication was blocked from new Cassandra node to old Cassandra nodes.