Solved: Re: APIGEEX External Token Verification, VerifyJWT...

mlan01

Hello all,

I am a first time poster. I have been experimenting with APIGEE for a month. One of my objectives is to build a proof of concept showing how to best use externally generated tokens using client_credentials grant in APIGEE. I have two demos working so far:

1. External with only verify (Simple)
Client reaches out to Keycloak to get the signed JWT token
Calls protected API on APIGEE with the APIGEE client_id for the app so we can identify who this is
VerifyJWT policy checks for signature, expiry, scopes etc...
Transaction is allowed to protected backend

2. External token trough service call and oAuth2 token generation (complex)
Client reaches out to APIGEE /get-token which triggers a service call to Keycloak to get JWT
Extract variables to get token from responce and set oauth_external_authorization_status = true
Client calls protected API with the APIGEE client_id (which is also the same in the keycloak client)
oAuth2 policy with VerifyAccessToken is triggered
Transaction is allowed to protected backend

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<DisplayName>OAuth-Store-External-Token</DisplayName>

<ExternalAccessToken>ext_access_token</ExternalAccessToken>

<Operation>GenerateAccessToken</Operation>

<Format>FORM_PARAM</Format>

</GenerateResponse>

<ReuseRefreshToken>false</ReuseRefreshToken>

<GrantType>client_credentials</GrantType>

</SupportedGrantTypes>

</OAuthV2>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<DisplayName>OAuthV2-verifyToken</DisplayName>

<Operation>VerifyAccessToken</Operation>

<Format>FORM_PARAM</Format>

</GenerateResponse>

<AccessToken>request.header.token</AccessToken>

</OAuthV2>

Questions:

What advantage do I get from approach 1 or 2? I understood that in scenario 2 the external token is stored in APIGEE token store for fast processing on the verifyaccesstoken operation?
It looks like the oAuth2 verifyAccessToken operation does not check the JWT signature? It looks like it resides on the fact that it has knowledge on that token since it was created by APIGEE and is stored in the token store. Is this correct?
When looking at the verify token policy output, what is the difference between the access_token and the apigee.access_token? it looks like APIGEE generated an opaque token? Why? The external token is eyJhbGc........

Thank you so much for any insight the community can provide.

dchiesa1

Apigee can issue tokens or verify OAuthV2 tokens.

For issuing, Apigee can issue OAuthV2 tokens in either "opaque" or JWT format. do this with the OAuthV2 policy, using Operation = GenerateAccessToken or GenerateJWTAccessToken, respectively. In either case these are "Apigee native" tokens. You can configure your proxies to later validate these tokens.
For validation of "Apigee native" tokens, you can use OAuthV2 policy with a different Operation - either VerifyAccessToken for opaque tokens, or VerifyJWTAccessToken for the JWT format.

But in many cases people want to include some external token issuer into the mix. Apigee can verify/validate a JWT that was generated by some external issuer, like KeyCloak or Entra ID (nee Azure AD) or Ping Identity or Okta, etc. Use VerifyJWT policy step type within Apigee to do this job.

Provide that policy with a source for the public key - usually this is a JWKS endpoint, but it could come in other forms - and the policy will:

verify that the JWT is wellformed (header, payload, signature)
verify that the source of public key can be used to verify the digital signature on the JWT
verify that the token is not expired, is "valid" (not being used before its "not before" timestamp)
verify the values of any other claims - issuer, audience, subject, or even custom claims

If VerifyJWT completes successfully , then your proxy can trust the contents of the JWT, and can evaluate the claims and perform conditional processing based on those claims. For example, if the externally-provided JWT contains a set of roles or scopes or groups associated to the Subject, you can configure your Apigee API proxy to do different things based on those values. Route things differently, allow or deny access to different subsets of APIs, and so on.

With all that, you can see that Apigee can validate a token , whether it was issued by Apigee, or issued by some external system. But there is a difference in Apigee behavior or function, comparing between these cases.

With the "Apigee native" token, Apigee will perform API-product based authorization checks. If you are familiar with the API Product concept in Apigee, you know that each product gets a configured set of operations. In the simple case, these are REST operations, like GET/products/1234 or POST /products (the latter request might be to create a new product). When your caller presents a token, it checks that the token is good for the API product, and then checks the current operation (GET, POST, DELETE, etc, and whatever URL path is presented), to see if the current operation is allowed by the cofngiured set of operations on the API Product. This is a clean, simple, slick way of performing an authorization check for your APIs. It is a way to operationalize the idea that not all tokens are good for all operations. That's just required if you have an API program with more than 2 or 3 resources. You get this behavior "automatically" when your API Proxy verifies an "Apigee native" token via the OAuthV2 policy step type.
With the externally-provided token, which, for the purposes of this conversation, we will assume is always a JWT format, Apigee just verifies the signature on the token, and maybe a few claims values. Upon success, your proxy can trust the values in the token. Apigee does not perform any API Product-based check with this kind of token. There is a way to get the API-product based authz check with an externally-issued token - and that is to follow the VerifyJWT with a subsequent invocation of VerifyAPIKey (screencast on VerifyAPIKey here: https://youtu.be/4FYZQuKuG3Y?si=7kbU9w1Q4fh1wknx) . In the standard pattern, you pull out one claim from the signed JWT, usually the clientid claim, or something like that, and use THAT as the input to VerifyAPIKey. If the client id you have configured in Keycloak (or whatever) is synchronized with the clientid in Apigee, then... you magically get the API Product check, when you invoke VerifyAPIKey.

So those are the two simple cases: Apigee native token, or externally-issued token. There are options to combine these. One of them is "token exchange", in which the client obtains a token from an external issuer , and then presents that token to Apigee in a request-for-access-token. To exchange one token for another. During the token exchange, your token-dispensing proxy in Apigee would verify the externally-supplied JWT and then issue an opaque token . And the result of that would be an "Apigee native" token.

Another option is to configure Apigee as a facade to the external token issuer. In which case you could build the token exchange implicitly into the OAuth flow. (screencast here: https://youtu.be/SlpqmkB6XVA?si=Ywwd3mgaRyE9kzka)

Why would you go to the trouble of issuing an opaque OAuth token in exchange for a JWT? You suggested it yourself: it's faster for Apigee to verify an opaque OAuth token. Verifying a signature is "pretty fast", like single digit milliseconds in latency, and we build in caching in Apigee to try to optimize that. But it still represents a cost to transmit the entire JWT, and handle it. An opaque token is smaller, faster to transmit, and much faster to validate (much less than 1ms) because it is completely cacheable.

But either way works. In most cases it won't really matter if it takes 4ms or 0.1ms to validate a token . The difference will not be detectable by the end user. This makes a difference at scale though, when you have tens of thousands of concurrent requests, and every millisecond counts.

View solution in original post

dchiesa1

Apigee can issue tokens or verify OAuthV2 tokens.

For issuing, Apigee can issue OAuthV2 tokens in either "opaque" or JWT format. do this with the OAuthV2 policy, using Operation = GenerateAccessToken or GenerateJWTAccessToken, respectively. In either case these are "Apigee native" tokens. You can configure your proxies to later validate these tokens.
For validation of "Apigee native" tokens, you can use OAuthV2 policy with a different Operation - either VerifyAccessToken for opaque tokens, or VerifyJWTAccessToken for the JWT format.

But in many cases people want to include some external token issuer into the mix. Apigee can verify/validate a JWT that was generated by some external issuer, like KeyCloak or Entra ID (nee Azure AD) or Ping Identity or Okta, etc. Use VerifyJWT policy step type within Apigee to do this job.

Provide that policy with a source for the public key - usually this is a JWKS endpoint, but it could come in other forms - and the policy will:

verify that the JWT is wellformed (header, payload, signature)
verify that the source of public key can be used to verify the digital signature on the JWT
verify that the token is not expired, is "valid" (not being used before its "not before" timestamp)
verify the values of any other claims - issuer, audience, subject, or even custom claims

If VerifyJWT completes successfully , then your proxy can trust the contents of the JWT, and can evaluate the claims and perform conditional processing based on those claims. For example, if the externally-provided JWT contains a set of roles or scopes or groups associated to the Subject, you can configure your Apigee API proxy to do different things based on those values. Route things differently, allow or deny access to different subsets of APIs, and so on.

With all that, you can see that Apigee can validate a token , whether it was issued by Apigee, or issued by some external system. But there is a difference in Apigee behavior or function, comparing between these cases.

With the "Apigee native" token, Apigee will perform API-product based authorization checks. If you are familiar with the API Product concept in Apigee, you know that each product gets a configured set of operations. In the simple case, these are REST operations, like GET/products/1234 or POST /products (the latter request might be to create a new product). When your caller presents a token, it checks that the token is good for the API product, and then checks the current operation (GET, POST, DELETE, etc, and whatever URL path is presented), to see if the current operation is allowed by the cofngiured set of operations on the API Product. This is a clean, simple, slick way of performing an authorization check for your APIs. It is a way to operationalize the idea that not all tokens are good for all operations. That's just required if you have an API program with more than 2 or 3 resources. You get this behavior "automatically" when your API Proxy verifies an "Apigee native" token via the OAuthV2 policy step type.
With the externally-provided token, which, for the purposes of this conversation, we will assume is always a JWT format, Apigee just verifies the signature on the token, and maybe a few claims values. Upon success, your proxy can trust the values in the token. Apigee does not perform any API Product-based check with this kind of token. There is a way to get the API-product based authz check with an externally-issued token - and that is to follow the VerifyJWT with a subsequent invocation of VerifyAPIKey (screencast on VerifyAPIKey here: https://youtu.be/4FYZQuKuG3Y?si=7kbU9w1Q4fh1wknx) . In the standard pattern, you pull out one claim from the signed JWT, usually the clientid claim, or something like that, and use THAT as the input to VerifyAPIKey. If the client id you have configured in Keycloak (or whatever) is synchronized with the clientid in Apigee, then... you magically get the API Product check, when you invoke VerifyAPIKey.

So those are the two simple cases: Apigee native token, or externally-issued token. There are options to combine these. One of them is "token exchange", in which the client obtains a token from an external issuer , and then presents that token to Apigee in a request-for-access-token. To exchange one token for another. During the token exchange, your token-dispensing proxy in Apigee would verify the externally-supplied JWT and then issue an opaque token . And the result of that would be an "Apigee native" token.

Another option is to configure Apigee as a facade to the external token issuer. In which case you could build the token exchange implicitly into the OAuth flow. (screencast here: https://youtu.be/SlpqmkB6XVA?si=Ywwd3mgaRyE9kzka)

Why would you go to the trouble of issuing an opaque OAuth token in exchange for a JWT? You suggested it yourself: it's faster for Apigee to verify an opaque OAuth token. Verifying a signature is "pretty fast", like single digit milliseconds in latency, and we build in caching in Apigee to try to optimize that. But it still represents a cost to transmit the entire JWT, and handle it. An opaque token is smaller, faster to transmit, and much faster to validate (much less than 1ms) because it is completely cacheable.

But either way works. In most cases it won't really matter if it takes 4ms or 0.1ms to validate a token . The difference will not be detectable by the end user. This makes a difference at scale though, when you have tens of thousands of concurrent requests, and every millisecond counts.

mlan01

Amazing, the great and powerful Dino is here. I just want to say thank you for responding. I have watched and read many of videos/posts. You are a true IT warrior.

Here are some additional test I did and follow-up questions:

Scenario 1 (VerifyJWT with additional VerifyAPIKey Policy with extracted key):
- Works as expected. I had to make sure the client_id was extracted
- Without VerifyAPIKey I guess the VerifyJWT policy will validate any JWT as long as it is signed and not expired. It has no notion of who the client is unless we add the verifyAPIKey and the client_id is known to APIGEE.
- This entire flow VerifyJWT + VerifyAPIKey took on avg: ~10ms
Scenario 2 (Oauth2 with GenrateToken External = YES and Store Token = NO)
- Wanted to see if what happens if I use oAUth2 policy to generate based on external token and set store token to false.
- The oAuth2 VerifyAccessToken fails, I am denied, and I believe it is because APIGEE has no reference to this token. It is not part of its Token Store.
Scenario 3 (OAuth2 with GenrateToken External = YES and Store Token = YES)
- Using client credentials from Keycloak that are synchronised to APIGEE to trigger oAuth2 GenerateAccessToken. JWT is stored in token cache
- oAuth2 VerifyAccessToken works.
- The oAuth2 VerifyAccessToken flow took on avg: ~20ms

Follow up questions:

1) Your statement regarding the JWTVerify makes wonder something. This is essentially useless correct? If we are not binding a token to a product, then how are we to even assert if the client_id is valid. Any JWT from my keycloak that is signed would work without the VerifyAPIKey policy part of the flow.

@dchiesa1 wrote:
With the externally-provided token, which is this telling is always a JWT format, Apigee just verifies the signature on the token, and maybe a few claims values. Upon success, your proxy can trust the values in the token. Apigee does not perform any API Product-based check with this kind of token

2) What drives a team to use the VerifyJWT policy VS the oAuth2 with VerifyAccessToken operation policy? My understanding is that a oAuth2 with VerifyAccessToken policy can only execute against a token that is part of the token store thus meaning I needed to run a oAuth2 with GenerateAccessToken operation prior to verify regardless if the token is external or internal. VerifyJWT is decoupled from the token store as it decodes at runtime.

What approach is better? What should an organization think about when choosing? The verifyJWT (with a follow up verifyAPIkey) feels simpler and performed well in my initial tests. I guess the difficulty is to keep the IDP and APIGEE client_id in synchronisation....

3) Your statement about token exchange. Why would this be desirable? Does this elevate the security? What would make me want to go down this path?

@dchiesa1 wrote:
One of them is "token exchange", in which the client obtains a token from an external issuer , and then presents that token to Apigee in a request-for-access-token. To exchange one token for another. During the token exchange, your token-dispensing proxy in Apigee would verify the externally-supplied JWT and then issue an opaque token . And the result of that would be an "Apigee native" token.

Thank you again

dchiesa1

Scenario 1 (VerifyJWT with additional VerifyAPIKey Policy with extracted key):

Works as expected. I had to make sure the client_id was extracted

Without VerifyAPIKey I guess the VerifyJWT policy will validate any JWT as long as it is signed and not expired. It has no notion of who the client is unless we add the verifyAPIKey and the client_id is known to APIGEE.
CORRECT.

This entire flow VerifyJWT + VerifyAPIKey took on avg: ~10ms
SOUNDS ABOUT RIGHT

==

Scenario 2 (Oauth2 with GenrateToken External = YES and Store Token = NO)

Wanted to see if what happens if I use oAUth2 policy to generate based on external token and set store token to false.

The oAuth2 VerifyAccessToken fails, I am denied, and I believe it is because APIGEE has no reference to this token. It is not part of its Token Store.
CORRECT.

==

Scenario 3 (OAuth2 with GenrateToken External = YES and Store Token = YES)

Using client credentials from Keycloak that are synchronised to APIGEE to trigger oAuth2 GenerateAccessToken. JWT is stored in token cache

oAuth2 VerifyAccessToken works.

The oAuth2 VerifyAccessToken flow took on avg: ~20ms
For the first one, that sounds about right. For subsequent calls, this data will be cached and it will be very fast.

@mlan01 wrote:

Your statement regarding the JWTVerify makes wonder something. This is essentially useless correct? If we are not binding a token to a product, then how are we to even assert if the client_id is valid. Any JWT from my keycloak that is signed would work without the VerifyAPIKey policy part of the flow.

Correct. With VerifyJWT you are establishing that "some other party" (in your case Keycloak) asserted the claims in the JWT. If you trust the third party, then your API proxy can trust the claims. The claims may or may not have any significance to Apigee. If you synchronize the client id, then the clientid in the third-party-provided JWT will be significant to Apigee, and you can invoke VerifyAPIKey on it to get product-based authorization checks.

@mlan01 wrote:

What approach is better? What should an organization think about when choosing? The verifyJWT (with a follow up verifyAPIkey) feels simpler and performed well in my initial tests. I guess the difficulty is to keep the IDP and APIGEE client_id in synchronisation....

Calling VerifyJWT each time through... doesn't necessarily take advantage of a cached result. There is *some* caching of JWTs seen by this policy, but..for sure, OAuthV2/VerifyAccessToken caches the token first time through, and then relies on the in-memory cache for satisfying subsequent verifications. It will perform very well.

As for which is better, it's up to you! Performance differences between 10 and 20ms are often immaterial when designing a system. It might be "nice" to have the API request be 10ms faster, but ... probably not at the cost of simplicitly or maintainability . It might be nice for me to be able to travel from 0-60 in 2.3s in my car, but not at the cost of $80,000 for a double-motor electric vehicle. Other people may make a different choice.

@mlan01 wrote:

Your statement about token exchange. Why would this be desirable? Does this elevate the security? What would make me want to go down this path?

A token exchange can perform the VerifyJWT ONCE, and then generate a new opaque access token (native to Apigee) and send it back to the client. Now the client sends a smaller opaque token with each request. It's faster to transmit and faster to validate. That's the advantage of adding the complexity of a token exchange.

One more note on that - OauthV2 is sort of magical in that... it allows and encourages a variety of token grant types. You can supporting different applications with different auth requirements, just by supporting different token generation approaches. Just different grant types. Regardless of the grant type used, the result is always.... a token. In the case of Apigee, we would generally recommend issuing an opaque access token in response, for any grant type.

That means you can keep the same OAuthV2/VerifyAccessToken in every proxy, even if you want to offer a multiplicity of grant types - authorizaiton code, password, client credentials, jwt-bearer, and so on. Grant the token any way you choose, but the verification (which happens a million times more often than the grant) is always the same. Always just VerifyAccessToken.

If I were designing a system for scale, across multiple different consumer types and application types, I would want that. I would want to exploit te variety of OAuthV2 grant types, while keeping verification very simple.

And that's all I have to say about that.

mlan01

Thank you so much for the interaction to my questions. I will keep watching your videos. Keep up the excellent work.

dchiesa1

@mlan01 wrote:

Amazing, the great and powerful Dino is here. I just want to say thank you for responding. I have watched and read many of videos/posts. You are a true IT warrior.

Ha! This made me laugh. I'm glad that you found my stuff helpful.

APIGEEX External Token Verification, VerifyJWT or Oauth2-VerifyAccessToken