Does message logging ( Cloud logging element) policy need the logger related pods to be running

[shegde]

For Apigee Hybrid, does Message Logging (Cloud Logging element) policy need the logger related pods to be running to be able to push the API logs to GCP?. We have currently set logger to false in our overrides and set up the cluster in AKS and want API logs to be available in GCP Cloud logging.

also we have used a shared flow with a service account mapped to push the logs

[apickelsimer]

Message Logging policy, including the Cloud Logging configuration, is unrelated to the logger pods. Thus, no need for the logger pods be present for the policy to work.

[shegde]

@apickelsimer  Thanks for the quick response. That is our understanding too. We recently set logger to false and not able to see logs in Cloud logging anymore. Not sure if it coincided with any other GCP level configuration. Hence wanted to check. Are there any pointers you could share around how to validate that it works ?

[apickelsimer · Accepted solution]

I'm not sure if there is any way to catch exceptions thrown by Message Logging policy when it fails to connect to the target logging service. Perhaps there is a way to debug this further but nothing obvious to me at the moment.

One other idea that comes to mind is to test your configuration (e.g. payload, service account credentials, etc) by invoking the Cloud Logging API directly. You could first start by doing this locally (i.e. via cURL) to make sure your parameters and credentials work as expected. Then, if that works, you could set up a Service Callout to do the same, using those same exact parameters. You can of course debug the Service Callout easier. This was the recommended way before Apigee added support for Cloud Logging natively in the MessageLogging policy. An example of the older approach using Service Callout can be found here.

If those both work as expected, I would then compare to the configuration you have for the CloudLogging element in your MessageLogging policy and make sure everything is the same.

[sravanipabathi]

Hi..

We are using Apigee Hybrid  v1.8...

For logging to GCP, I tried using the approach of Cloud logging through service callout - its working in proxy postflow but not in post client flow(https://github.com/DinoChiesa/Apigee-GCP-Logging-Example)

Also tried using MessageLogging policy - it is not logging to GCP logs

(https://cloud.google.com/apigee/docs/api-platform/reference/policies/message-logging-policy#cloudlog...)

FYI - Using the same service account credentials for both the approaches and it has 'roles/logging.admin' role

Please advise if there is anyway to troubleshoot why we are not able to view the logs in GCP using MessageLogging policy -Cloud-logging approach.

Thanks !

[dknezic]

Just to check, do you just have the one policy in the post client flow or have you moved multiple policies across?

[sravanipabathi]

hi,

For the first approach of leveraging Dino's article https://github.com/DinoChiesa/Apigee-GCP-Logging-Example,

I implemented as per the article ... It works in proxy postflow, but not in postclientflow(I don't see any response headers once it is moved to post-client flow)

The second approach - leveraging built-in 'MessageLogging' policy, its not working(not sending logs to GCP) either in proxy postflow or postclientflow ... For this, I have just 1 policy 'MessageLogging' in the postclientflow.

First approach : Logging policies using service-callout in proxy postflow - displays the variables and it works

First approach: Logging policies using service callout in post-client flow : doesn't work(not fetching data from KVM)

Second approach: leveraging builtin 'MessageLogging' policy in post-client flow:

[sravanipabathi]

@dknezic/@apickelsimer  Would appreciate any reply.. Thank you !

[dknezic]

When you get a chance, please respond to my previous question and/or share your proxy xml config

[sravanipabathi]

@dknezic : I replied in the above post with the policies and screenshots I am using in 2 approaches:

Also please find the proxy xml config below for the 2nd approach using only 'Messagelogging' policy:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
<PreFlow name="PreFlow">
<Request/>
<Response/>
</PreFlow>
<Flows/>
<PostFlow name="PostFlow">
<Request/>
<Response/>
</PostFlow>
<PostClientFlow>
<Response>
<Step>
<Name>Message-Logging-2</Name>
</Step>
</Response>
</PostClientFlow>
<HTTPProxyConnection>
<BasePath>/test-hybrid-proxy</BasePath>
</HTTPProxyConnection>
<RouteRule name="default">
<TargetEndpoint>default</TargetEndpoint>
</RouteRule>
</ProxyEndpoint>

messageLogging policy config:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging continueOnError="false" enabled="true" name="Message-Logging-2">
<DisplayName>Message Logging-2</DisplayName>
<CloudLogging>
<LogName>projects/{organization.name}/logs/apigee-{apiproxy.name}</LogName>
<Message contentType="application/json">{"x-request-id": "{messageid}",
"organization": "{organization.name}",
"environment": "{environment.name}",
"apiProxyName": "{apiproxy.name}",
"apiProxyRevision": "{apiproxy.revision}",
"request": {"uri": "{request.uri}","method":"{request.verb}"},
"response":{"statusCode":"{message.status.code}"}
}</Message>
<ResourceType>api</ResourceType>
</CloudLogging>
<logLevel>INFO</logLevel>
</MessageLogging>

[dknezic]

Your trace seems to show other policies as well? (Other policies could prevent policies from running in post client flow or could be manipulating the response or other variables leading to missing data)

[sravanipabathi]

@dknezic/@apickelsimer : In second approach, I added 'Messagelogging' policy inside shared flow in postclientflow.. No other policies are present in the shared flow.. Also I tried with only 'Messagelogging' policy in postclientflow instead of keeping it inside shared flow.. that also did not work !

Is there any other thing that could be prevent Messagelogging in sending logs to cloud logging explorer?

[sravanipabathi]

@dknezic/@apickelsimer

This is my proxy endpoint xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
<PreFlow name="PreFlow">
<Request/>
<Response/>
</PreFlow>
<Flows/>
<PostFlow name="PostFlow">
<Request/>
<Response/>
</PostFlow>
<PostClientFlow name="PostClientFlow">
<Response>
<Step>
<Name>Message-Logging-2</Name>
</Step>
</Response>
</PostClientFlow>
<HTTPProxyConnection>
<BasePath>/test-hybrid-proxy</BasePath>
</HTTPProxyConnection>
<RouteRule name="default">
<TargetEndpoint>default</TargetEndpoint>
</RouteRule>
</ProxyEndpoint>

This is the 'MessageLogging' policy xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging continueOnError="false" enabled="true" name="Message-Logging-2">
<DisplayName>Message Logging-2</DisplayName>
<CloudLogging>
<LogName>projects/{organization.name}/logs/apigee-logs</LogName>
<Message contentType="application/json">{"x-request-id": "{messageid}",
"organization": "{organization.name}",
"environment": "{environment.name}",
"apiProxyName": "{apiproxy.name}",
"apiProxyRevision": "{apiproxy.revision}",
"request": {"uri": "{request.uri}","method":"{request.verb}"},
"response":{"statusCode":"{message.status.code}"}
}</Message>
<ResourceType>api</ResourceType>
</CloudLogging>
<logLevel>INFO</logLevel>
</MessageLogging>

[kishore-hl]

Runtime SA must be impersonated with proxy SA as stated below. It worked for me.

https://cloud.google.com/apigee/docs/api-platform/security/google-auth/overview#api-proxy-configurat...