Failed to flush writer error on Cassandra

pranjalic
Participant II

I am not able to start the Cassandra instance. It is throwing below error while checking the Cassandra status :

apigee-service apigee-cassandra status

opt/apigee/apigee-service-4.17.05-0.0.572/bin/../lib/service-functions.sh: fork: retry: Resource temporarily unavailable

/opt/apigee/apigee-service-4.17.05-0.0.572/bin/../lib/service-functions.sh: fork: retry: Resource temporarily unavailable

Also in the Cassandra log I can see below error :

Failed to flush writer .no space left on device.

We do have enough disk space on the machine . Not sure what's causing this error.

Also should it affect the Edge-UI. As we are not able to access UI.

0 7 831
7 REPLIES 7

Not applicable

Can you share more information on the size of your node?
How many cassandra nodes do you have (size of your ring)?
Are the other C* nodes good and is there any difference between these nodes? @Pranjali

I get this info on my org:

/opt/apigee/apigee-cassandra-2.1.16-0.0.1049/bin/nodetool status
Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN 10.54.16.174 713.93 MB ? 4beb1271-****-****-****-e6b54031134e 0 ra-1

Not sure how to get other details.

Also, paste the output of

1. `df -h`

2. sudo lsblk -l

3. value for `data_file_directories` from cassandra.yaml file

4. value for `commitlog_directory` from cassandra.yaml file

Hi Vaibhav,

Please see the details below:

1. `df -h`

5400-df-h.jpg

2. sudo lsblk -l

As I don't have root access, can not execute this command.

3. value for `data_file_directories` from cassandra.yaml file\

data_file_directories:

/opt/apigee/data/apigee-cassandra/data

4. value for `commitlog_directory` from cassandra.yaml file

commitlog_directory:

/opt/apigee/data/apigee-cassandra/commitlog

Hi @vaibhav :

Please see the details below :

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-root
5.8G 946M 4.6G 17% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 976M 175M 735M 20% /boot
/dev/mapper/rootvg-home
240M 3.6M 220M 2% /home
/dev/mapper/rootvg-opt
13G 8.5G 3.2G 73% /opt
/dev/mapper/rootvg-osmf
3.9G 182M 3.5G 5% /osmf/mgmt
/dev/mapper/rootvg-scheduler
213M 2.1M 196M 2% /osmf/mgmt/scheduler
/dev/mapper/rootvg-usr
4.8G 1.4G 3.2G 31% /usr
/dev/mapper/rootvg-var
5.8G 1.1G 4.4G 21% /var
none 1.0G 21M 1004M 3% /tmp
/dev/mapper/appvg-applv1
9.7G 2.6G 6.6G 28% /oraclient
/dev/mapper/appvg-applv2
20G 2.7G 16G 15% /software/bea
/dev/mapper/appvg-applv3
40G 3.2G 34G 9% /wls_domains
/dev/mapper/appvg-applv5
25G 44M 24G 1% /software/bea_hwb2brtp1
/dev/mapper/appvg-applv6
25G 44M 24G 1% /wls_domains/hwb2brtp1
/dev/mapper/appvg-applv7
30G 44M 28G 1% /wls_domains/hwb2brtp1_logs
/dev/mapper/appvg-applv8
30G 44M 28G 1% /wls_domains/hwb2brtp1_filestores
/dev/mapper/appvg-applv4
20G 12G 7.3G 62% /wls_domains_logs

sudo lsblk -l
Password:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 35G 0 disk
sda1 8:1 0 1G 0 part /boot
sda2 8:2 0 34G 0 part
rootvg-root (dm-0) 252:0 0 6G 0 lvm /
rootvg-osmf (dm-11) 252:11 0 4G 0 lvm /osmf/mgmt
rootvg-scheduler (dm-12) 252:12 0 224M 0 lvm /osmf/mgmt/scheduler
rootvg-opt (dm-13) 252:13 0 12.5G 0 lvm /opt
rootvg-usr (dm-14) 252:14 0 5G 0 lvm /usr
rootvg-var (dm-15) 252:15 0 6G 0 lvm /var
rootvg-home (dm-16) 252:16 0 256M 0 lvm /home
sdb 8:16 0 5G 0 disk
swapvg-swap (dm-1) 252:1 0 5G 0 lvm [SWAP]
sdc 8:32 0 215G 0 disk
sdc1 8:33 0 215G 0 part
appvg-applv1 (dm-2) 252:2 0 9.9G 0 lvm /oraclient
appvg-applv2 (dm-3) 252:3 0 19.9G 0 lvm /software/bea
appvg-applv3 (dm-4) 252:4 0 39.9G 0 lvm /wls_domains
appvg-applv4 (dm-5) 252:5 0 20G 0 lvm /wls_domains_logs
appvg-applv5 (dm-6) 252:6 0 24.9G 0 lvm /software/bea_hwb2brtp1
appvg-applv6 (dm-7) 252:7 0 24.9G 0 lvm /wls_domains/hwb2brtp1
appvg-applv7 (dm-8) 252:8 0 29.9G 0 lvm /wls_domains/hwb2brtp1_logs
appvg-applv8 (dm-9) 252:9 0 29.9G 0 lvm /wls_domains/hwb2brtp1_filestores
appvg-applv9 (dm-10) 252:10 0 14.9G 0 lvm /appl/b2b/support
sr0 11:0 1 1024M 0 rom

less /opt/apigee/apigee-cassandra-2.1.16-0.0.1049/conf/cassandra.yaml

commitlog_directory: /opt/apigee/data/apigee-cassandra/commitlog

data_file_directories - /opt/apigee/data/apigee-cassandra/data

please let me know if you need more information.

Hi Pranjali

add to /etc/security/limits.conf file (you need root or sudo to edit it) the following lines:

apigee soft memlock unlimited 
apigee hard memlock unlimited 
apigee soft nofile 32768 
apigee hard nofile 65536 
apigee soft as unlimited 
apigee hard as unlimited
apigee - nproc 32768

Also restart cassandra. Report back your new findings !

Will help you if this not fully resolves and we will also look at available disk space on partition.

Basically the error message says, Cassandra cannot find enough disk space to flush the memtables to sstables into the disk.

In the df -h output, could you help us identify which is the data and commitlog directory for Cassandra ?

typically data directory is /opt/apigee/data/apigee-cassandra/data

Alternatively, you can do this

cd /opt/apigee/data/apigee-cassandra/data
df -h .

Check for available disk space