How to update ROUTER logs (nginx access log) from apigee policy?

[#1]

How to update MESSAGE PROCESSOR logs using messageContext or executionContext object?

I am looking to update/cutomize message processor logs in apigee using messageContext or executionContext object. I find examples where they use javascript to send customized logs with the help of Splunk or third party logging system url. But i do not have access to my log server url, so have to update the logs through message context or execution context object. Can someone help me find a solution ?

Let me explain my requirement better.
I have my logs getting updated at

source = /opt/apigee/var/log/edge-router/nginx/marsh~Dev-US.9001_access_log

which is the message processor logs. I want to add more info to this log message from apigee. I see there is a message logging policy, but i am not sure how to use it to update my infomraiton to this log.

currently my log looks like this:

2018-06-27T08:52:03-05:00 usdf23v0488 10.23.119.33:62390 10.23.132.186:8998 0.179 - - 200 200 511 255 POST /proxy/m2-digital-broker-api-1.0.0-SNAPSHOT/auth/client/accesstoken HTTP/1.1 usdf23v0488-29638-119015-1 PostmanRuntime/7.1.1 10.23.132.186 usdf23v0488-29638-119015-1 10.23.119.33 - - - - - - green -1 /organizations/marsh/environments/Dev-US/apiproxies/M2-Digital-Broker-API-1.0.0-SNAPSHOT /proxy/m2-digital-broker-api-1.0.0-SNAPSHOT/auth/client/accesstoken dc-1 gateway green

I want to add more info to this log. Can you help me how to do this. Below is the message logging policy i used in my post flow

<MessageLogging name="Message-Logging-1">
  <DisplayName>Message Logging-1</DisplayName>
  <Syslog>
    <Message>HELLOHELLOHELLO</Message>
  </Syslog>
</MessageLogging>

@ Dino-at-google- can you please help me with this ?

[dchiesa1]

What are you trying to log? Give more details about the information you would like to log.

Are you using Apigee Edge OPDK ?

When you say 'message context' and 'execution context' object, are you referring to aJava Callout? It seems there are a bunch of details you have left out from your question.

[dchiesa1 · Accepted solution]

First, you said that

source = /opt/apigee/var/log/edge-router/nginx/marsh~Dev-US.9001_access_log

...is the message processor log. This is not true. This log file is for nginx. The router. it logs inbound API requests to the system.

Using the MessageLogging policy, you can log to a different, independent file.

This policy does not allow you to log to the nginx log file.

Also, to log to a file, you need to use the File element, not Syslog. Use the Syslog element when you have a Syslog endpoint.

Details are described in the fine documentation. The doc tells you exactly where the log file will appear. (Ignore the video there - i just looked and it's irrelevant to your question)

[#4]

//

This policy does not allow you to log to the nginx log file.

Also, to log to a file, you need to use the File element, not Syslog. Use the Syslog element when you have a Syslog endpoint.//

Thanks for your response Dino. So you say that i SHOULD NOT use MessageLogging Policy to log into nginx log.
Can you help me how to log my custom information into naginx log file. which policy to use ?


What i tried BELOW IS MY POLICY

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="true" enabled="true" name="Message-Logging-1">
<DisplayName>Message Logging-1</DisplayName>
<File>
<Message>This is a test message. Message id : {request.header.messageId}</Message>
<FileName>/opt/apigee/var/log/edge-router/nginx/marsh~Dev-US.9001_access_log</FileName>
<FileRotationOptions rotateFileOnStartup="true">
<FileRotationType>SIZE</FileRotationType>
<MaxFileSizeInMB>10</MaxFileSizeInMB>
<MaxFilesToRetain>10</MaxFilesToRetain>
</FileRotationOptions>
</File>
</MessageLogging>

But still my log does not show my message:BELOW IS THE LOG

2018-06-28T00:14:13-05:00 usdf23v0488 10.23.119.33:50348 10.23.132.186:8998 0.017 - - 401 401 481 46 POST /proxy/m2-digital-broker-api-1.0.0-SNAPSHOT/auth/client/accesstoken HTTP/1.1 usdf23v0488-53466-148804-1 PostmanRuntime/7.1.1 10.23.132.186 usdf23v0488-53466-148804-1 10.23.119.33 false policy invalid_request oauthV2/GenerateAccessToken Authentication /organizations/marsh/environments/Dev-US/apiproxies/M2-Digital-Broker-API-1.0.0-SNAPSHOT green -1 /organizations/marsh/environments/Dev-US/apiproxies/M2-Digital-Broker-API-1.0.0-SNAPSHOT /proxy/m2-digital-broker-api-1.0.0-SNAPSHOT/auth/client/accesstoken dc-1 gateway green

• host = usdf23v0488
• source = /opt/apigee/var/log/edge-router/nginx/marsh~Dev-US.9001_access_log
• sourcetype = nginx:plus:access

Can you help me figure out how to update this "nginx" access log, from my apigee policy.

[dchiesa1]

So you say that i SHOULD NOT use MessageLogging Policy to log into nginx log.

Incorrect. I say you CANNOT.

Can you help me how to log my custom information into naginx log file. which policy to use ?

Nope. You can't. There's no supported way to do that. It's also probably a bad idea.

If you want to consolidate the nginx log with the message processor log, then you should use a log aggregator, like StackDriver or Splunk or similar.

[#6]

@Dino-at-Google Hi Dino, I have replied below for your answer with my POLICY and LOG details. Can you please help me how to resolve it.

I am not getting the message (i give in my policy ) in my router log. I replaced the policy with "File" element as you mentioned, But still i dont see it working.

[dchiesa1]

When using MessageLogging with the FileName element, You cannot specify the directory location of the logfile, via configuration in the policy. You must specify the base file name only.

(In my opinion, it's a bug that the policy allows you to specify a path. Also the documentation doesn't make this clear either. I've raised a doc bug on that.)

Instead of this:

<FileName>/opt/apigee/var/log/edge-router/nginx/marsh~Dev-US.9001_access_log</FileName><br>

Try this:

<FileName>marsh~Dev-US.9001_access_log</FileName><br>

If you want to set the location of the MP log files, the documentation I cited previously describes how. Please see that doc for more information!

[#8]

@Dino-at-Google Thanks Dino for the reply. But even after changing i get the log like below only
2018-06-28T15:34:27-05:00 usdf23v0488 10.23.119.33:54715 10.23.132.184:8998 0.015 - - 401 401 481 46 POST /proxy/m2-digital-broker-api-1.0.0-SNAPSHOT/auth/client/accesstoken HTTP/1.1 usdf23v0488-109982-178474-1 PostmanRuntime/7.1.1 10.23.132.186 usdf23v0488-109982-178474-1 10.23.119.33 false policy invalid_request oauthV2/GenerateAccessToken Authentication /organizations/marsh/environments/Dev-US/apiproxies/M2-Digital-Broker-API-1.0.0-SNAPSHOT green -1 /organizations/marsh/environments/Dev-US/apiproxies/M2-Digital-Broker-API-1.0.0-SNAPSHOT /proxy/m2-digital-broker-api-1.0.0-SNAPSHOT/auth/client/accesstoken dc-1 gateway green

• host = usdf23v0488
• source = /opt/apigee/var/log/edge-router/nginx/marsh~Dev-US.9001_access_log
• sourcetype = nginx:plus:access
  
  
  I do not find my message in the log. Below is my policy
  
  <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  <MessageLogging async="false" continueOnError="true" enabled="true" name="Message-Logging-1">
  <DisplayName>Message Logging-1</DisplayName>
  <File>
  <Message>This is a test message. Message id : {request.header.messageId}</Message>
  <FileName>marsh~Dev-US.9001_access_log</FileName>
  <FileRotationOptions rotateFileOnStartup="true">
  <FileRotationType>SIZE</FileRotationType>
  <MaxFileSizeInMB>10</MaxFileSizeInMB>
  <MaxFilesToRetain>10</MaxFilesToRetain>
  </FileRotationOptions>
  </File>
  </MessageLogging>
  
  
  Am i making any basic issue with my approach of trying to update my router log ?

[dchiesa1]

am i making any basic issue with my approach of trying to update my router log ?

We are experiencing what is known as a "communications issue".

Let me try to clarify again. Please read this list carefully, 3 times:

• You cannot use the MessageLogging Policy to log into the nginx log.
• You can specify the filename (not the path!) in the MessageLogging policy
• there is a default path for the logfile written by the MessageLogging policy
• the documentation gives information on how to set the location to something else
• if you try to set that location to the nginx log, something will break
• if you want aggregated logs, use a log aggregator

[#10]

Thanks Dino for the reply. You wrote:

• You cannot use the MessageLogging Policy to log into the nginx log.
• if you try to set that location to the nginx log, something will break
• if you want aggregated logs, use a log aggregator

a few more questions

- if messageLoggingPolicy cannot log into nginx log. how else i can add my message to nginx logs ?
- is it even possible to add my message to the nginx logs ???
- is log aggregator the answer to my question on adding messages to the existing nginx logs ???

[dchiesa1]

You cannot configure the MP to write to the nginx log. Nginx writes to the nginx log.

if you want aggregated logs, use a log aggregator. You can try Apache Kafka or Apache Flume for this purpose.