Hybrid - Best place to save JWT keys.

[Giupo]

Hi,

Hello everyone, I have seen the documentation which states that sensitive data must be saved in the K8s Secrets, but does this also apply to the encryption keys of the JWTs? I would think that it is better to use KVMs because they are encrypted and nobody can access them.

In terms of performance, however, I think it is better to use the K8s secrets instead of consuming Cassandra's transactions.

What do you recommend?

Thanks in advance

[dchiesa1 · Accepted solution]

It's good to be mindful of performance when designing distributed systems. 

If you use KVM to store the crypto keys, the keys are prevented from disclosure, and you will incur the I/O cost of reading from a persistent store - in hybrid it happens to be Cassandra. However the KVM is also backed by a cache, which means you can amortize the cost of the I/O across many uses of the key.  

Also keep in mind that the crypto operations to sign, encrypt, verify, or decrypt a JWT also incur CPU time. There's a cost to just "doing JWTs", regardless of where you store the keys. It can take several ms to run the cryptography. This can be comparable to the time required for key retrieval. Therefore optimizing _just_ for key retrieval may be insufficient. 

With all this in mind, in my experience using the eKVM is a good approach to storing keys, and, amortized across many transactions, the I/O contributes a relatively low cost to the overall cost of handling (generating or verifying) JWTs. 

You didn't ask, but let's talk about an optimization that has a higher potential payoff that changing where you store the keys....

Consider a system in which the Apigee API proxy verification of a signed JWT for every request for service from a client.  The proxy might use the eKVM to store the key, with a reasonable cache TTL, like maybe 10 minutes.  The cache is fast, just a memory lookup, so key retrieval costs you approximately 0ms for 99.999% of requests.  Verification will take 5ms or so, each request. 

Compare that to a system in which the client, when initiating a conversation, exchanges a signed JWT for an opaque Apigee-issued OAuth token. We can call this OAuth token exchange. and then the client uses the opaque token on subsequent requests for service.  There is a cost in the beginning to verify the JWT (~5ms) and a cost to issuing an OAuth token (maybe 5ms). On subsequent requests, the FIRST time the opaque OAuth token is presented, there is an I/O cost to retrieving the token from Cassandra. Again, maybe 5ms.  And then subsequently, there is a near-zero cost to verifying the opaque token, because it is in cache.  Approximately 0ms.

So if you have a case in which a client will invoke an API ~100 times in the course of a "conversation", the first system, that simply uses signed JWT, will incur a flat cost of around 5ms for each request, just for authentication.  The second system will, on average, incur a near-zero cost for authentication. 

JWT are cool, and also , using JWT implies consuming CPU cycles during generation and verification.  Use them wisely! 

[Giupo]

Very helpful thanks!

My requirement is for every request that arrives on Apigee to route the traffic to the backend by attaching a JWT, as the Keycloak is enabled on the backend, which validates the JWT to authorize the request coming from the Gateway.

From what I have understood, it is therefore advisable to specify on the KeyValueMapOperations policy a <ExpiryTimeInSecs>  with a value of a few minutes in order to read the key value from the L1 cache, without consuming the I/O on Cassandra.

At this point you would recommend this approach too for technical credentials, instead of using secrets? from what has emerged it seems to me a better approach, right?

Thanks in advance

[dchiesa1]

it is therefore advisable to specify on the KeyValueMapOperations policy a <ExpiryTimeInSecs> with a value of a few minutes in order to read the key value from the L1 cache, without consuming the I/O on Cassandra.

Yes, but use ExpiryTimeInSeconds, which is a new element, which replaces the older ...InSecs element. It works the same way.

At this point you would recommend this approach too for technical credentials, instead of using secrets?

Yes I do recommend the use of JWT, rather than sending secrets over the wire. But I am not sure I am understanding and addressing your question. If possible it would be nice to use transport-layer security - TLS - between Apigee and the upstream system.  Maybe in lieu of application-layer security (a JWT that gets validated by Keycloak) or in addition to application layer security. 

[Giupo]

Sorry for my english Dino, I meant:

At this point, you would recommend this approach (KVM) also for technical credentials (username:password), instead of using K8s secrets to store them?

Because in doc I had red "Use Secrets to store sensitive data such as user credentials", but I prefer your approach using KVM.

[Giupo]

I want to add another aspect to consider (from here), which may be of help to others in the future:

Consider using Kubernetes Secrets when:

• You need to store sensitive information that you never want to be stored in the management plane.