Atomic KVM update

[emarcotte]

Hi,

I frequently see recommendations to hide API Proxy configuration in KVM entries so that they can be deployed to different targets and not have to mutate the proxy definition. For instance maybe putting an "Audience" field in the KVM so a development target can vary from production.

I also see that there is no "atomic" KVM entry update API as observed in a previous thread,but there is an "override" in the KVM Operations available to proxies. Is the override option atomic?

If there is no atomic option, I'm curious to understand what the recommend approach to 0 downtime deployments/KVM updates is. Do I have to make new KVM entries (e.g. value_v1, value_v2, value_v3) or something and rotate the proxy to use the new value as I make updates somehow?

Thanks for any tips/tricks/hints,

Eugene

[dchiesa1 · Accepted solution]

I guess by the statement there is no "atomic" KVM entry update, you mean that the current API does not allow updates (PATCH/PUT). Is that right?

If so, let me first confirm the premise of the question. In Apigee X / hybrid, it is not possible to use the administrative API to perform a PUT (update) on an existing KVM entry.  

## KVM - update entry for one keyvaluemap - THIS IS NOT SUPPORTED - returns 404
:endpoint = https://apigee.googleapis.com

PUT :endpoint/v1/organizations/:org/environments/:env/keyvaluemaps/:kvm/entries/:entryname
Authorization: Bearer :token
Content-Type: application/json

{"name":":entryname", "value":"value-for-entry"}

---

@emarcotte wrote:

I'm curious to understand what the recommend approach to 0 downtime deployments/KVM updates is.

---

I believe get your point. I believe your question is, what happens if, between the Delete/Write, the KeyValueMapOperations policy reads the persistent store?  Presumably the policy will retrieve an empty value. Neither the old value, nor the new one, but ... nothing. 

First, what is the exposure?   I think the window of exposure is pretty narrow.  Not zero, but small, and maybe this mitigates some of your concern.  The KVM reads are cached, so each time you use KeyValueMapOperations to read (GET), the policy will first look in the cache to satisfy the read. Only if the cache is cold will the policy try to read from persistent store. So the "returns nothing" behavior will happen if and only if the cache expires just at the right time, between the delete and the write. 

Second, the API proxy should have protections for the case of "returns nothing" in any case. For example in the case of a misconfiguration, irrespective of the non-atomic update scenario.  Maybe a RaiseFault policy checked with a Condition on the retrieved value. The result at runtime is that you may have some number of API calls that return a 500 to the caller, during the narrow window. 

The workaround you propose is a reasonable one - use the KeyValueMapOperations / PUT with override. This will be atomic. 

Keep in mind that the KVM read cache is not distributed. So you may have some message processors returning the old value, and some returning the new value, until the cache expires. But in all cases they'll be able to read a value. 

---

@emarcotte wrote:

Do I have to make new KVM entries (e.g. value_v1, value_v2, value_v3) or something and rotate the proxy to use the new value as I make updates somehow?

---

I don't think that would work, because :rotate the proxy to use the new value: is more or less equivalent to the original problem. 

My advice

1. consider whether you need the atomic update. Depending on the frequency of change of KVM values, the lifetime of the KVM cache you specify (ExpiryTimeInSecs), the load on the proxy, and whether you have multi-region deployments, you may see more or less of a problem.  Also be sure you have a "catch" for empty value retrieved, and return a proper fault message. Maybe also log this condition and examine your logs to see how often it occurs. 
2. If you need it, use the KeyValueMapOperations policy itself. A good starting point is this kvm-admin proxy. It uses KeyValueMapOperations with PUT and override.

[emarcotte]

Thanks, lots to chew on.