Storing credentials,sensitive config - KVM v/s Vault

[#1]

Update: 2021 March 30

This article is out of date. It refers to two elements of the Apigee platform that have changed since this post was written.

• The Key-Value Map, a persistent store
• The Apigee Vault, another persistent store

In 2015, this post was accurate and helpful. In 2020, it is no longer accurate.

• the Apigee Vault is no longer supported. It was part of the nodejs target, which has been deprecated, and is out of support since 2018 (I can't remember when exactly).
• The KVM now supports encryption.

As regards KVM Encryption, in Apigee Edge, you can configure your KVM to be encrypted. In Apigee X, all KVM are encrypted.

So, in summary: use KVM to store sensitive data. In Apigee Edge you can encrypt it. In Apigee X, it is always encrypted, with your key.

I've left the original post and comments for posterity's sake.

---

ORIGINAL POST

I have had this conversation with some of my peers on which approach to take while storing the credentials and other sensitive configuration which can be accessed by the runtime api in a secure and a performant way. The words "secure" and "performant" are important since we would want to have them both.

Two top choices would clearly be KVM and Apigee-Vault(Secure store service). Lets discuss about each and try come up with a more viable solution based on the use case.

The KVM way -

KVM's have been used in many,many implementations for storing environment related configurations like backend url's, usernames, passwords etc.

Pros -

1) Out of the box policy to GET,PUT or DELETE a KVM resource(KVM operations)

2) Performant, it was(may be is) known to have cached the KVM resources for some time, which can be super performant if almost every api call is performing the KVM operation.

3) You can have one place to store all your configurations, making it super easy for operations to maintain it.

4) Can be scoped all the way from org,env,proxy all the way through policy.

Cons -

1) Content is NOT encrypted by itself while storing data and is stored as plain text(However I think is should be a simple step to encrypt the data before storing it)

2) If not used correctly, say if it is used to store tremendously large data, then it could have an implication on the runtime traffic.

The Vault way -

Pros -

1)Data is encrypted and not stored as plain text!!

Cons -

1) No out of the box policy to retrieve the data - Needs a method apigee.getVault() which can be accessed through node.js

2) Configs stored in different places - You might not want all your configuration to be store in the Vault and would end up storing some either in config.json or KVM, which could be difficult to maintain.

3) Not necessarily a con but unlike KVM Vaults can only be scoped to an org or an environment level.

[#2]

While Vault is more difficult to use, it's almost certainly the best way to store sensitive data such as credentials. Two facts make me believe this: 1) Vault is encrypted, but other potential methods are not, and 2) Practically speaking, information stored in Vault can only be *retrieved* by a runtime proxy making it more difficult for unauthorized personnel to gain access, where KVM can be accessed at runtime *and* via the management API. Non-sensitive data can be stored in many other places (KVM and custom attributes on the API Product, Developer or Developer Application come to mind most often), but credentials really should be encrypted when stored and access should be controlled as closely as possible.

Even though Vaults are scoped to org or environment level, you can have many entries at each level. This gives you a roughly equivalent level of flexibility to that provided by proxy-scoped KVM entries.

[#3]

@Chris von See Very valid point. However, my concern is this - Since the only method of retrieving the vault variables is through a node.js implementation and currently node.js can only be implemented as a target and not a callout, this might impact my analytics.

Say if I have to grab the credentials for almost every api call hitting a target, How will I get the correct analytics with what you are suggesting? @David Allen

[#4]

@Vinit Mehta You're correct in saying that target latency metrics won't be accurate because the actual target call may not be made from the endpoint defined in the TargetEndpoint. Other analytics - call counts, overall resource request latency, error codes, etc. are still valid.

Calling your target endpoint from Node.js might be one way to address the point you raise.

[#5]

@Chris von See Just to clarify - when you say other analytics are still valid that is assuming you are calling the actual target as well from node.js, correct?

[ceber]

Vault is definitely the way to go. There's an enhancement request to enable access to vault storage from the regular proxy flow. IMO that's required, so hopefully the primary con will go away.

[#7]

No, what I mean is that even if you call the real "target" via a ServiceCallout policy after the script target is called your overall request latency numbers (but not the target latency numbers) will still be accurate. I thought perhaps calling the target from Node.js after retrieving the creds from Vault might give you more accurate target latency stats as well.

[mdunker]

Nice write up, @Vinit Mehta.

I believe KVMs can be used safely, but it is strongly recommended to lock them down via RBAC and encrypt data that is put in them, decrypting in the proxy. I feel that there is a cost to the use of Vault at this time, potentially forcing users to use node if they aren't using it otherwise.

That being said, as soon as Vault is available via policy, it will surely be the preferred way.

[#9]

Nice point @Mike Dunker , one more advantage of using KVM is that it will be cached in Mp and lookup can be fast where as Vault if believe its on Cassandra but not sure if we have cache enabled for that . any idea @Vinit Mehta ?

[sarthak]

@Maruti Chand I do not think Vault has any cache enabled in front of it.

[ozanseymen]

I don't think we can pass any security review if we store plain text passwords in C*. We also need to think about private cloud here as original article didn't make any distinctions.

Another approach is to create a new proxy with a node.js target just to get the data from vault. Original proxy will then cache the data for x hours. Not the best approach but seems good enough until we get policy access to vault... This option obviously requires protection around this vault access api.

[#12]

Just curious, is this still an enhancement request or is work already completed/underway to create a Vault-Access sort of policy?

[jdsa]

Yes. Work is underway. Please stay tuned.

[#14]

Is there somewhere that we can look for updates on this?

[jdsa]

Nowhere public but we should have something towards the end of April.

[#16]

Forgot to ask - will it also be available for on-prem as well in April? or a future release? (we are on-prem)

[jdsa]

All our features roll out to the cloud first and are then automatically available in the next on-prem release. We are in an advanced date of development and are hoping to go live around the April timeframe. Things might change just a bit. If things look good in the cloud, they will definitely be in the following on-prem release.

[mchalmers]

What difference - if any - in auditing / logging capability between these methods? How granular is the logging around access to KVM via Management APIs - e.g. in case of a key change actions? Does audit logging differ by user type?

[ozanseymen]

@mchalmers - we only audit log create/update/delete actions on developers, users, organisations, api products, api proxies and apps - currently. So KVM and vault get/set operations are not audit logged.

[#20]

Hi @Joel D'sa

Any update on this feature? Has it been already released in the cloud? What are the plans for on-prem release? Thank you in advance!

[williamking]

@Joel D'sa I'm looking to access vault values from within a policy (as opposed to Node). Is this feature available in cloud deployment? Haven't found it.

[jdsa]

Yeah sorry guys. This is a feature ready to go but we need to reprioritize shipping it in favor of some other features. This should be in production middle of August. I will definitely update this thread if that date changes.

[#23]

Hey - it's mid-August now - any word on this feature? We're looking forward to it!

[jdsa]

The feature is complete but it's not quite production ready yet. Would you want to give it a spin in a beta environment? We would love some feedback.

[akoo]

Hello all, I wanted to add an important note: encrypted KVMs are here. Details are in our documentation: http://docs.apigee.com/api-services/reference/key-value-map-operations-policy . You now have an option for encrypted data without having to use Node.js.

[jdsa]

http://apigee.com/about/blog/engineering/storing-sensitive-data-encrypted-kvms