VerifyJWT failure with JWT tokens issued by Apigee

[salonzo]

We have implemented OpenID Connect authentication for some API Proxies with the following steps :

• Authentication flow :
  1. User identity information is stored into a JWT token generated with a GenerateJWT policy (kid and private key are coming from  a KVM in the Environment Configuration)
     

     <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
     <GenerateJWT name="JWT.GenerateIdToken.RS256">
         <Algorithm>RS256</Algorithm>
         <PrivateKey>
             <Value ref="private.privatekey"/>
             <Id ref="kid"/>
         </PrivateKey>
         <Id/>
         <AdditionalClaims ref="IT-customClaimsArrayJSON"/>
         <OutputVariable>output.id-jwt</OutputVariable>
     </GenerateJWT>

  2. An opaque accesstoken is generated with a OAuthV2 policy and the JWT token is attached as an attribute
     

     <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
     <OAuthV2 async="false" continueOnError="false" enabled="true" name="OA.GenerateAccessTokenOpaque">
         <Operation>GenerateAccessToken</Operation>
         <Attributes>
             <Attribute name="jwt_access_token" ref="output.id-jwt" display="false"/>
         </Attributes>
     </OAuthV2>

  3. The opaque token is returned to the consumer to be used as Bearer token for the subsequent API calls

• API call flow
  1. Opaque token is verified using OAuthV2 policy
     

     <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
     <OAuthV2 async="false" continueOnError="false" enabled="true" name="OA.VerifyToken">
         <Operation>VerifyAccessToken</Operation>
         <AccessTokenPrefix>Bearer</AccessTokenPrefix>
     </OAuthV2>

  2. JWT token attached to the opaque token is verified with a VerifyJWT policy to extract user identity (JWKS used is coming from KVM in the Environment Configuration)
     

     <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
     <VerifyJWT async="false" continueOnError="false" enabled="true" name="JWT.VerifyAccessToken.JWKS">
         <Algorithm>RS256</Algorithm>
         <Source>accesstoken.jwt_access_token</Source>
         <PublicKey>
             <JWKS ref="oidc.jwks"/>
         </PublicKey>
         <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
     </VerifyJWT>

  3. The call is forwarded to the backend with user identity information

This flow is working fine in general but we have a few error cases in production that can hardly be explained : the opaque token verification succeeds (which means the consumer application has been properly authenticated) but the JWT signature fails (which should never happen since the token has been issued by apigee using the same keys as for the verification)

When that happens, the VerifyJWT policy fails with a 401 error and 'steps.jwt.InvalidToken' message

{
    "fault": {
        "faultstring": "Invalid token: policy(JWT.VerifyAccessToken.JWKS)",
        "detail": {
            "errorcode": "steps.jwt.InvalidToken"
        }
    }
}

Since the error rate is very low (0.1% of the requests) it means it cannot come from the code or the configuration (or else it would fail much more frequently).
Yet, it's still happening regularly (almost every day) and it's not linked to any configuration change since the kid, private key and JWKS have not changed in months (we rotate these keys once per year)

As this happens mostly in production and apparently randomly it's very hard to reproduce it during a trace session, so we have no clue what could be causing such an issue.

Note: we are using Apigee Edge on-premise

[salonzo]

I've been able to add more information in the logs and it seems that when this VerifyJWT failure occurs it's because the jwt_access_token attribute of the opaque token (set in step 2 of the Authentication flow described in the previous post) has an empty value.

Which is different than if the attribute was missing because in this case the VerifyJWT policy would fail with a different message (in step 2 of the API call flow described in the previous post).
Instead of "steps.jwt.InvalidToken" the error would be "steps.jwt.FailedToResolveVariable"

So either the issue was with the OAuthV2 policy during the opaque token generation or it's in  the OAuthV2 policy during the opaque token verification.
In both cases it looks like it could be related with this issue #18559 since in all cases the Authentication flow and the API call flow were processed by different MessageProcessors.

[dchiesa1]

I suggest that you open a case with the Apigee support team; they may have more information about the other issue. It seems like you've done some diagnosis to narrow down the problem scenario.  That will be helpful.