Stackdriver extension in Apigee X?

In Apigee Edge, we are using Stackdriver extension to log the transactional logging. As Apigee X does not support extensions, what will be the best approach to log in GCP Cloud Logging from Apigee X API Proxy?

Solved Solved
0 12 1,530
2 ACCEPTED SOLUTIONS

What problem are you having?

The example you are following uses a private key, associated to a service account, to sign a JWT and then send it to google to request a token. In my experience the most common problem is that when using the GenerateJWT policy, the privatekey is misformatted. This happens when the key material provided is not in a format that can be parsed. To troubleshoot this, I suggest that you first embed the PEM-encoded key into an AssignMessage policy, like this:

<AssignMessage name='AM-SetPrivateKeyVariable'>
  <AssignVariable> 
    <Name>private.privatekey</Name> <!-- must use private prefix here -->
    <Value>
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCrXfjCAfJR8K+9
2rvB67ZyaGRdvSn3+NZdcEOxl0IV4f7YK5PockjcQVmaTmj6XfS4X8OTWi8EbdLZ
yaVa9CVL7ex7XhGXifoqUZW2o93fPPm015JGLei8LZhMIxLYma7hrGqh19kk9mwI
...
oBnc5zrx3gR1tTtj8rqkK/E=
-----END PRIVATE KEY-----
    </Value>
  </AssignVariable>
</AssignMessage>

And then refer to that variable in the policy configuration, as in

<GenerateJWT name='GJWT-1'>
 <PrivateKey>
  <Value ref='private.privatekey'/>
 </PrivateKey>
 ...

You need to insure that the AssignMessage is attached so that it executes before the GenerateJWT policy.

It is not possible for you to directly specify the PEM-encoded private key within the GenerateJWT policy; validation of the policy will prevent you from embedding a secret (the private key) inside GenerateJWT. But if you use AssignMessage to set the variable, that passes validation.

The one common challenge with this is, the Service Account JSON file encodes the private key in a form like this:

"-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC1uwMZRLIHg3+p\nvydbrbavESq+SLAFWy8T8...o7cpY/zXBoQ==\n-----END PRIVATE KEY-----\n" 

...which does not look very much like the multi-line string I showed above in the AssignMessage. The reason is: all of the newlines stored in the JSON file are encoded as \n - backslash-n. So you would need to use a text editor or some other mechanism to un-encode those newlines - essentially remove the outer quote marks, and then replace each of the \n sequences with an actual newline. Then insert the result of THAT into the AssignMessage policy I showed above.

Once that works, THEN you can try loading the same private key string into the KVM. And change your policy to refer to the variable you retrieve from KVM. Something like this:

<GenerateJWT name='GJWT-1'>
 <PrivateKey>
  <Value ref='private.extracted_private_key'/>
 </PrivateKey>
 ...

I typically do not load JUST the private key into the KVM. Rather, I load the entire service account key .json file into the KVM. And then within the API proxy, shred that JSON file to extract its various properties. This gist shows how I do that.

At that point, the client id, the private key, the oauth token endpoint, and other claims from that JSON file are all available as context variables.

If you could provide more detail as to what is going wrong, we might be able to help.

View solution in original post

Quick update: Sending logs from Apigee to Cloud Logging has become a lot easier thanks to the built-in Google Service Account Authentication that is now available in Apigee X.

For a simple example, please see this reference: https://github.com/apigee/devrel/blob/main/references/cloud-logging-shared-flow/sharedflowbundle/pol...

View solution in original post

12 REPLIES 12

fyi... I refered https://github.com/DinoChiesa/Apigee-GCP-Logging-Example But I am facing issues in configure/assign the Service Account's private key pem value in Apigee X KVM

If you're having issues storing the value in the KVM, you could create an administrative api proxy for interacting with your KVM as described here

https://github.com/apigee/devrel/tree/main/references/kvm-admin-api

You could then use this to store your service account details in an encrypted KVM

What problem are you having?

The example you are following uses a private key, associated to a service account, to sign a JWT and then send it to google to request a token. In my experience the most common problem is that when using the GenerateJWT policy, the privatekey is misformatted. This happens when the key material provided is not in a format that can be parsed. To troubleshoot this, I suggest that you first embed the PEM-encoded key into an AssignMessage policy, like this:

<AssignMessage name='AM-SetPrivateKeyVariable'>
  <AssignVariable> 
    <Name>private.privatekey</Name> <!-- must use private prefix here -->
    <Value>
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCrXfjCAfJR8K+9
2rvB67ZyaGRdvSn3+NZdcEOxl0IV4f7YK5PockjcQVmaTmj6XfS4X8OTWi8EbdLZ
yaVa9CVL7ex7XhGXifoqUZW2o93fPPm015JGLei8LZhMIxLYma7hrGqh19kk9mwI
...
oBnc5zrx3gR1tTtj8rqkK/E=
-----END PRIVATE KEY-----
    </Value>
  </AssignVariable>
</AssignMessage>

And then refer to that variable in the policy configuration, as in

<GenerateJWT name='GJWT-1'>
 <PrivateKey>
  <Value ref='private.privatekey'/>
 </PrivateKey>
 ...

You need to insure that the AssignMessage is attached so that it executes before the GenerateJWT policy.

It is not possible for you to directly specify the PEM-encoded private key within the GenerateJWT policy; validation of the policy will prevent you from embedding a secret (the private key) inside GenerateJWT. But if you use AssignMessage to set the variable, that passes validation.

The one common challenge with this is, the Service Account JSON file encodes the private key in a form like this:

"-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC1uwMZRLIHg3+p\nvydbrbavESq+SLAFWy8T8...o7cpY/zXBoQ==\n-----END PRIVATE KEY-----\n" 

...which does not look very much like the multi-line string I showed above in the AssignMessage. The reason is: all of the newlines stored in the JSON file are encoded as \n - backslash-n. So you would need to use a text editor or some other mechanism to un-encode those newlines - essentially remove the outer quote marks, and then replace each of the \n sequences with an actual newline. Then insert the result of THAT into the AssignMessage policy I showed above.

Once that works, THEN you can try loading the same private key string into the KVM. And change your policy to refer to the variable you retrieve from KVM. Something like this:

<GenerateJWT name='GJWT-1'>
 <PrivateKey>
  <Value ref='private.extracted_private_key'/>
 </PrivateKey>
 ...

I typically do not load JUST the private key into the KVM. Rather, I load the entire service account key .json file into the KVM. And then within the API proxy, shred that JSON file to extract its various properties. This gist shows how I do that.

At that point, the client id, the private key, the oauth token endpoint, and other claims from that JSON file are all available as context variables.

If you could provide more detail as to what is going wrong, we might be able to help.

Hi @Dino-at-Google

Thanks for your response.

fyi...

I tried couple of options (Apigee X KVM, Assigned Message), but currently the below one (by JS) is working for me

// just take the private key value (as it is) from SA JSON file and assigned here
var pkey = "-----BEGIN PRIVATE KEY-----\nMIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDOYj1xj3xO/0nm\nOL889G/Rv8OfELMA6RjJt1iDtW77IKXnrD03WQa2RAZmojoIUxO7/FbnDmRWkyVb\nP2uHmT5G8qBmk8295NM7+1dwPeIK2M8srEZ5bZTgi/0/ep2dUg0CeObQX2M30iYP\naczgvIa...................==\n-----END PRIVATE KEY-----\n";

print("pkey :"+pkey);

context.setVariable("private.privateKey",pkey);

And using this flow variable in Generate JWT plocy as below

<GenerateJWT continueOnError="false" enabled="true" name="JWT-Generate-GCP-Logging-token">
    <DisplayName>JWT-Generate-GCP-Logging-token</DisplayName>
    <Algorithm>RS256</Algorithm>
    <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
    <PrivateKey>
        <Value ref="private.privateKey"/>
    </PrivateKey>
    <Issuer ref="private.gcplogging.jwt_issuer"/>
    <Audience>https://www.googleapis.com/oauth2/v4/token</Audience>
    <ExpiresIn>90s</ExpiresIn>
    <AdditionalClaims>
        <Claim name="scope">https://www.googleapis.com/auth/logging.write</Claim>
    </AdditionalClaims>
    <OutputVariable>gcp_bearer_jwt</OutputVariable>
</GenerateJWT>

Quick update: Sending logs from Apigee to Cloud Logging has become a lot easier thanks to the built-in Google Service Account Authentication that is now available in Apigee X.

For a simple example, please see this reference: https://github.com/apigee/devrel/blob/main/references/cloud-logging-shared-flow/sharedflowbundle/pol...

So much simpler! 

@strebel @dchiesa1 I tried to use the code given in above link for logging in Apigee X shared flow.

I am using Maven Plugin and CICD pipeline. I have  given Log writer permission to service account which is running the pipeline as well. I am getting below error.

 

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.3.0:deploy (deploy-bundle) on project <shared_flow_name>: : MojoExecutionException: com.google.api.client.http.HttpResponseException: 400 Bad Request
[ERROR] {
[ERROR] "error": {
[ERROR] "code": 400,
[ERROR] "message": "deployment validations failed",
[ERROR] "status": "FAILED_PRECONDITION",
[ERROR] "details": [
[ERROR] {
[ERROR] "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
[ERROR] "violations": [
[ERROR] {
[ERROR] "type": "MISSING_SERVICE_ACCOUNT",
[ERROR] "subject": "organizations/<org_name>/sharedflows/<shared_flow_name>/revisions/19",
[ERROR] "description": "Deployment of \"organizations/<org_name>/sharedflows/<shared_flow_name>/revisions/19\" requires a service account identity, but one was not provided with the request."
[ERROR] }

 

Do I need to explicitly provide the Service Account name in service callout ?

If yes , how do I do it ?

 

There are conceptually two service accounts here:

  1. The SA used for deploying an API proxy. This one needs permissions to create a revision and deploy it to a certain environment.
  2. The Service account used by a proxy deployment to authenticate against Google APIS. To configure it on the mvn deploy plugin you set the `apigee.googletoken.email` property and ensure that SA 1 (from above) has the permission to `actAs` SA 2 (this one).

The service account is not explicitly mentioned in the service callout but you just add a Google Authentication that is then using the SA that is associated with the deployment.

Yes. And see also, this recent Q&A. And this one.

When using the maven plugin,

  • Use -Dfile to specify a .json file for the SA to use to invoke Apigee APIs (the #1 from strebel's reply)
  • use -DgoogleTokenEmail to specify #2.

Thank You 

Thank You for the response 

Quick update on this topic, with Apigee X the Message Logging policy directly supports writing to Cloud Logging (f.k.a. Stack Driver).

https://cloud.google.com/apigee/docs/api-platform/reference/policies/message-logging-policy#cloudlog...