We recently had an issue case of cache data handling on some private cloud customer who is using PopulateCache policy with KeyFragment which will add new cache entries.
We need to carefully manage the numbers of newly added cache entries by the KeyFragment reference if the source is dynamically changed and deleted by the expiration of the cache.
It will impact on heap memory consumption on message processor having Full GC frequently and results in API request failure as well as high CPU load. You can monitor it with some jstat metrics like FGC count/time, etc.
Also the L2 cache on Cassandra nodes creates so many tombstones on its deletion of the cache expired and add/remove cache entries causing resource impacts on memory and disk usage - accompanied by Memtable flush and SSTable compaction for these data on cassandra nodes also the data will be replicated to other nodes on writing/clearing data which can induce traffic surge and can lead to a failure in repair sessions.
This is what we observed and I hope this will help other users with the similar use case.
Thanks,
Toshi
LinkedIn
Twitter