Inconsistent results from KVM

[#1]

We are seeing strange results from KVM lookups. We are updating the value for a key multiple times in a short period of time (once every second or so for 5 updates). When we query the KVM from the management server API it's 100% accurate. But if we query from a policy we are seeing varying results. Sometimes it returns the correct value but often it returns an old value. This only appears to be the case when multiple MPs are running. And the same MP returns different results over time, so it does not appear to be a strict caching issue. We did see that one message process of the three seemed to always return the correct response. Maybe that's the one the MS is hitting??? We are perplexed--any input is appreciated. We are running on-prem.

[#2]

Hello,

KVM's are cached internally for some minutes (or secs) for performance. You might want to retry after say 2-3 minutes and it should give you consistent updated results form the api calls (policy execution) we well.

[#3]

@Vinit Mehta thanks for the response. Where is the caching, on the message processor or in cassandra? We thought it might be the message processor but we see inconsistent results from the same message processor over time. So are there multiple caches on the same MP? Does the management server not cache? The results from that always seem up-to-date.

[#4]

@Jeff Nadeau - Yes that little KVM caching happens at the MP level. Since the run time traffic is on MP, it caches there. No it would not be cached at the Management Sever, that is why you always see consistent results there. However its not normal to see the same MP give you different results, are you sure it gave you results like old,new,old..?Did you check after say like 2-3 mins? Was the runtime traffic still fetching old values?

[#5]

Yes, @Vinit Mehta, I am sure. We have the traffic going over 3 MPs and we were watching the logs on which one it hit and what the results were. The number is the MP number and X means wrong answer and C means correct answer. Here is what we saw over multiple hits: 1X 1C 2C 1X 1C 1C 2C 1C 3X 1X 1C 2C 1X 1C 3C 1X 1C 2C 2C 1X 2C 1C 3X 2C 1C 1C 2C 1C then mostly correct until we saw a 3X then all correct. (It's interesting the MP #2 always gave correct response.) This was over more like 5 minutes than 2-3. But, yes, it resolves itself. Any idea why this behavior? Is the cache configurable? Thanks.

[#6]

@Vinit Mehta, do you have any more info on this? I submitted a support ticket (875436), but any additional info you have on MP caching of KVM values would be much appreciated.

[#7]

@Jeff Nadeau As I mentioned earlier that KVM's are cached at MP for performance reasons. So an update can take upto a couple of minutes to reflect on your runtime traffic. HOWEVER it is not normal to see such inconsistencies within the same MP. You have done the right thing by opening a support ticket, I would follow up on that. BTW its still not that bad as you mentioned that everything works as expected in 2-3 minutes.

[#8]

@Vinit Mehta

Yes, understand that aspect of it. Thanks for the info. But it would be bad if there were no way for me to change the caching behavior. If I am making a critical update to the KVM then 2-3 minutes may be too long to wait. I should be able to make the performance-vs-accuracy tradeoff decision and not have it made for me.

[asribalaji]

@Jeff Nadeau Whats the value set for ExpiryTimeInSecs in your kvm policy ? If its -1, could you try setting it to 5/10 seconds & check the behavior of the key value after the update . thanks

<KeyValueMapOperations async="false" continueOnError="false" enabled="true" name="CreateFooKVM" mapIdentifier="FooKVM">
  <DisplayName>CreateFooKVM</DisplayName>
  <ExpiryTimeInSecs>-1</ExpiryTimeInSecs>