Cassandra disks filling very fast

Not applicable

Hi Apigee Team,


We have an issue for our cassandra nodes, where disk space are filling very fast on 3 nodes out of 6 nodes. All these nodes are in one data center. We see followings are main directories which are utilizing almost all used space:-

vgbx21vr:apigee:/var/SP/apigee-data/apigee-cassandra/data $ df -h /var/SP
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg1-varSPvol 60G 53G 4.4G 93% /var/SP
vgbx21vr:apigee:/var/SP/apigee-data/apigee-cassandra/data $


vgbx21vr:apigee:/var/SP/apigee-data/apigee-cassandra/data $ du -sh *|grep G
13G apprepo
11G cache
24G kms
vgbx21vr:apigee:/var/SP/apigee-data/apigee-cassandra/data $

If you notice above, 48 GB is consumed by above 3 directories out of 53 GB. We already have below scripts scheduled to run on cassandra nodes -

vgbx21vr
1 0 * * 1 PATH=/opt/SP/java/default/bin:$PATH; /opt/apigee/apigee-cassandra/bin/nodetool -h `hostname -i` clearsnapshot; /opt/apigee/apigee-cassandra/bin/nodetool repair -pr;
vgbx22vr
1 0 * * 2 PATH=/opt/SP/java/default/bin:$PATH; /opt/apigee/apigee-cassandra/bin/nodetool -h `hostname -i` clearsnapshot; /opt/apigee/apigee-cassandra/bin/nodetool repair -pr;
vgbx23vr
1 0 * * 3 PATH=/opt/SP/java/default/bin:$PATH; /opt/apigee/apigee-cassandra/bin/nodetool -h `hostname -i` clearsnapshot; /opt/apigee/apigee-cassandra/bin/nodetool repair -pr;
vgbx24vr
1 0 * * 4 PATH=/opt/SP/java/default/bin:$PATH; /opt/apigee/apigee-cassandra/bin/nodetool -h `hostname -i` clearsnapshot; /opt/apigee/apigee-cassandra/bin/nodetool repair -pr;
vgbx25vr
1 0 * * 5 PATH=/opt/SP/java/default/bin:$PATH; /opt/apigee/apigee-cassandra/bin/nodetool -h `hostname -i` clearsnapshot; /opt/apigee/apigee-cassandra/bin/nodetool repair -pr;
vgbx26vr
1 0 * * 6 PATH=/opt/SP/java/default/bin:$PATH; /opt/apigee/apigee-cassandra/bin/nodetool -h `hostname -i` clearsnapshot; /opt/apigee/apigee-cassandra/bin/nodetool repair -pr;


Could you please let us know what could be the reason and best possible approach for investigation to start ?

Thanks

Vinod

0 0 353
0 REPLIES 0