Getting "connection refused" error while logging into Cassandra

[adevegowda]

When we are trying to login to Cassandra on Private Cloud version 4.16.05, we are getting the following error:

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

[adevegowda · Accepted solution]

To understand the cause for "Connection refused" error, tried out the following steps:

1. Ran the netstat command and got the following output:

[root@adevegowda bin]# 
netstat -an | grep 9042 tcp 0 0 ::ffff:10.1.22.10:9042 :::* LISTEN 

The above output indicated that we are listening on port 9042 and IP address 10.1.22.10.

2. However, the error clearly showed that cqlsh was trying to connect to 127.0.0.1 by default. So we were getting "connection refused" error.

3. Modified the cqlsh command to connect to the IP address 10.1.22.10 and re-ran it as follows and we were able to connect to Cassandra database successfully.

[root@adevegowda bin]# sudo /opt/apigee/apigee-cassandra/bin/cqlsh 10.1.22.10 9042 -u cassandra -p cassandra 
Connected to Apigee at 10.1.22.10:9042. 
[cqlsh 5.0.1 | Cassandra 2.1.13 | CQL spec 3.2.1 | Native protocol v3] 
Use HELP for help. 
cassandra@cqlsh> 

In Summary, the command is:

sudo /opt/apigee/apigee-cassandra/bin/cqlsh <IP-address> <port#> -u cassandra -p cassandra 

[DChiesa]

Hmmm, it looks like it is expecting to find a Cassandra node on the same machine as the management server runs on. And it is failing to connect. Maybe this is an All-in-one Edge install.

I would start diagnosing by:

1. examining the status of the C* process, and other processes
   /opt/apigee/apigee-service/bin/apigee-all status
   IF the process is not running, then that would be the reason for the connection error.
2. Check the log file of Cassandra. you should find it in :
   /opt/apigee/var/log/apigee-cassandra/system.log
   There was a bug in Edge 16.05 prior to patch 05 where the system.log was not THERE.
   Let me know and I'll tell you how to find it. if you are on 1701 then this problem won't affect you.
   The system.log for C* may provide some clues for why C* was not running, or was not accepting connections.
   
3. Check the standard things on the machine, like... available disk space, available memory, and so on. If you are running All-in-one, do you have enough virtual memory? (how much?) etc.

Good luck!

[adevegowda]

@Dino,

Thanks for your suggestions.

[#5]

I was doing research on - how to make Cassandra listen on multiple interfaces eg. eth0 and loopback for remote calls. You might stump into this situation when you want to introduce a 'local tcp proxy' in the Cassandra nodes to proxy 'remote client communications'.

Here is what I found -

You could set rpc_address to 0.0.0.0 (Note: 'rpc_address' does not allows to specify multiple interface IPs, hence you have to use 0.0.0.0 and make cassandra listen on all interfaces).

Set the broadcast_rpc_address to <ip_address_of_the_node> (In most of the cases it will be IP of eth0, this is needed because if broadcast_rpc_address is not set it will by default pick rpc_address)

If there is a better way to achive this please let me know.