MessageLogging Policy (CloudLogging) - Set higher scope fields

[bselistre]

Hello, 

For performance reasons, I am currently trying to change my existing logging flow to use the MessageLogging policy instead of the ServiceCallout version (https://github.com/apigee/devrel/tree/main/references/cloud-logging-shared-flow).

Using ServiceCallout, I was able to set fields outside jsonPayload object (e.g. severity, httpRequest, etc.)

"entries": [{
        "severity": "INFO",
         "httpRequest": {
           "requestMethod": "{request.verb}",
           "requestSize": "{requestSize}",
           "remoteIp": "{client.ip}",
           "requestUrl": "{proxy.url}",
           "status": "{message.status.code}",
           "userAgent": "{userAgent}"
        },
        "jsonPayload": {
            "organization": "{organization.name}",
            "environment": "{environment.name}",
            "apiProxy": "{apiproxy.name}",
            "apiProxyRevision": "{apiproxy.revision}",
            "apiProduct": "{apiproduct.name}",
            "developerApp": "{apiproduct.name}",
            "clientId": "{client_id}",
            "developerId": "{developer.id}",
            "requestUri": "{request.uri}",
            "requestUrl": "{request.url}",
            "verb": "{request.verb}",
            "correlationId": "{messageid}",
            "proxyRequestReceived": "{client.received.end.timestamp}",
            "proxyResponseSent": "{client.sent.end.timestamp}",
            "targetResponseReceived": "{target.received.end.timestamp}",
            "targetRequestSent": "{target.sent.end.timestamp}",
            "targetResponseCode": "{message.status.code}",
            "proxyResponseCode": "{response.status.code}",
            "clientReceived": "{client.received.start.timestamp}",
            "clientSent": "{client.sent.start.timestamp}",
            "faultName":"{fault.name}"
        }
    }]

Unfortunately, MessageLogging policy only allow me to set fields inside jsonPayload object. So if the same payload as above is sent inside the <Message> tag, I will finish with something like this in the logs:

"jsonPayload": {
    "entries": [
        {
            "severity": "INFO",
            "httpRequest": {
                "requestMethod": "{request.verb}",
                "requestSize": "{requestSize}",
                "remoteIp": "{client.ip}",
                "requestUrl": "{proxy.url}",
                "status": "{message.status.code}",
                "userAgent": "{userAgent}"
            },
            "jsonPayload": {
                "organization": "{organization.name}",
                "environment": "{environment.name}",
                "apiProxy": "{apiproxy.name}",
                "apiProxyRevision": "{apiproxy.revision}",
                "apiProduct": "{apiproduct.name}",
                "developerApp": "{apiproduct.name}",
                "clientId": "{client_id}",
                "developerId": "{developer.id}",
                "requestUri": "{request.uri}",
                "requestUrl": "{request.url}",
                "verb": "{request.verb}",
                "correlationId": "{messageid}",
                "proxyRequestReceived": "{client.received.end.timestamp}",
                "proxyResponseSent": "{client.sent.end.timestamp}",
                "targetResponseReceived": "{target.received.end.timestamp}",
                "targetRequestSent": "{target.sent.end.timestamp}",
                "targetResponseCode": "{message.status.code}",
                "proxyResponseCode": "{response.status.code}",
                "clientReceived": "{client.received.start.timestamp}",
                "clientSent": "{client.sent.start.timestamp}",
                "faultName": "{fault.name}"
            }
        }
    ]
}

I have also tried adding custom fields inside the <Label> tag but it doesn't support message templating, so variables are treated as a hard-coded string.

The ServiceCallout option adds ~100ms in the total response time when compared to MessageLogging policy, even without the <Response> tag, which is not viable in some of our use cases.

So, I would like to know if there's any work-around that can be done to use MessageLogging policy and also have the additional fields in the log.

Thank you.

[dchiesa1 · Accepted solution]

The ServiceCallout option adds ~100ms in the total response time when compared to MessageLogging policy

Have you attached the ServiceCallout in the PostClientFlow ? It should not impact the response time the client observes, if you do it that way.

I would like to know if there's any work-around that can be done to use MessageLogging policy and also have the additional fields in the log.

I think the main problem is to avoid the added latency right? Attaching SC in the right place should do that.

[Manju]

Hi Dino,

Can we use ServiceCallout in the PostClientFlow ?

I tried this sharedflow https://github.com/g-lalevee/Apigee-X-GCP-Logging in Proxy Endpoint PostFlow. Works fine for success responses.

When the proxy fails, this flow is getting skipped, so I moved the shareflow to PostClientFlow. Eventhough in debug , I'm able to see sharedflow being executed but not able to see variables info and also logs are not going to GCP.

Thanks

[dchiesa1]

Can we use ServiceCallout in the PostClientFlow ?

Yes. There was a time when ServiceCallout was not permitted in PostClientFlow, but I believe that restriction was lifted for just this purpose. What version of Apigee are you using? There are a few different varieties: OPDK, Apigee Edge, Apigee X, Apigee hybrid... If you're on a current version of any of those, ServiceCallout should work as I suggested in PostClientFlow.

in debug , I'm able to see sharedflow being executed but not able to see variables info and also logs are not going to GCP.

That sounds frustrating. "Not able to see variables info" - I don't know what you mean there. But I understand that the logs are not getting sent to GCP.

Are you using response.content and other response.XXX variables in the ServiceCallout payload? If so that could be the reason that you are not seeing the values you expect, and also the reason the logs are not getting sent as you expect. Can you switch to message.content and so on? Use the message. prefix for variables. Alternatively you can build the json payload to be logged in the PostFlow by setting a new variable (with AssignMessage or JavaScript etc) and then log via ServiceCallout in the PostClientFlow, referencing that new payload variable. The construction of the payload happens in the PostFlow, when all the request and response variables are available. But it won't contribute much to latency, it involves only memory manipulation, no I/O, so it will be fast.

[bselistre]

In fact using the ServiceCallout in the PostClientFlow reduced significatively the added latency. Thank you @dchiesa1 

[sravanipabathi]

Hi bselistre,

I am using Apigee hybrid v1.8 and tried Cloud logging approach as mentioned in this articlehttps://github.com/DinoChiesa/Apigee-GCP-Logging-Example,

I also have this problem of 'sharedflow being executed but not able to see variables info and also logs are not going to GCP.' 

Sharedflow works in proxy postflow but not in postclient flow.. As the Sharedflow has KVM policies to retrieve GCP service account keys, fetching data from KVM is failing in postclient flow.. please let me know how is your problem fixed !

Appreciate any help in this ! Thanks in advance  !