One of the best ways to track down problems in APIs is through logging. The easiest way to do that In Apigee Edge is to attach a Message Logging policy on your API and log custom messages to a local disk or to syslog. When configured for syslog, it will forward log messages from Apigee Edge to a remote syslog server including external log management services, such a Splunk, Sumo Logic, and Loggly.
This article covers some of the common methods to log into Loggly from Apigee Edge. If you are interested in logging to Splunk, then jump to this article.
Pre-requisites
You should have an account with Loggly. I am using my free trial account for creating this article. In this post I will discuss two approaches to build this integration. However both the approaches require you to have an unique Consumer Token. You can get your Consumer Key from the Loggly management UI, under Source Setup tab.
Approach 1 : Using Message Logging Policy
You can attach a Message Logging policy either on the request or response path of your API. A simple Message Logging policy will look like this -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-1"> <DisplayName>Message Logging-1</DisplayName> <Syslog> <Message>[YOUR_LOGGLY_CONSUMER_TOKEN@41058 tag="{organization.name}.{apiproxy.name}.{environment.name}"] MESSAGE_YOU_WANT_TO_LOG </Message> <Host>logs-01.loggly.com</Host> <Port>514</Port> </Syslog> </MessageLogging>
Let's take a closer look at this policy, especially the section under <Syslog>.
<Message> : Represents the syslog message to be logged. This includes the header, tag (s) and the actual content. Loggly only accepts syslog events with a RFC5424 compliant header. Within the header there must be a structured data set that includes your Customer Token and the Loggly private enterprise number (PEN) which is 41058 (a constant). Tags are great for adding organization to your log events to aid in segmentation & filtering. They are metadata you can set on the source side, and can be included with any event that is transmitted to Loggly.
<Host> & <Port> : Loggly provides two syslog endpoints that accept both UDP and TCP syslog. For standard syslog please use:
logs-01.loggly.com:514
For secure syslog encrypted with TLS please use:
logs-01.loggly.com:6514.
In the Message Logging policy, these are represented under <Host> and <Port> sections.
Approach 2 : Using ServiceCallout Policy
Loggly provides standard HTTP based API interfaces for sending data. You can send a single line, multiple lines or bulk event data using these interfaces. With Apigee Edge, you can call this interface using a Service callout policy. A sample policy will look like this -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ServiceCallout async="false" continueOnError="false" enabled="true" name="Service-Callout-1"> <DisplayName>Service Callout-1</DisplayName> <Properties/> <Request variable="myRequest"> <Set> <Headers> <Header name="content-type">text/plain</Header> </Headers> <Payload> MESSAGE_YOU_WANT_TO_LOG </Payload> </Set> </Request> <Response>calloutResponse</Response> <HTTPTargetConnection> <Properties/> <URL>http://logs-01.loggly.com/inputs/YOUR_LOGGLY_CONSUMER_TOKEN/tag/http/</URL> </HTTPTargetConnection> </ServiceCallout>
NOTE : While using these interfaces you don't have to append **@41058** to your Consumer Token.
Viewing Logs in Loggly
The logs in Loggly can be viewed either on the Dashboard or by searching based on the tag.
NOTE : This article focus only on Apigee Edge and Loggly integration. Please refer to Loggly documentation if you need additional information of their capabilities.
References -
UPDATE :
We have fixed issue with Loggly message formatting while using message logging policy, for more details please refer article here.
I have implemented logging as described in this entry (both approach 1 and 2). However when I use the Message Logging Policy (approach1) I do not get the desired result in Loggly as with approach 2 using the service callout policy. With the message logging policy the content of the message is not recognized and resolved in Loggly as it is with the service callout policy. I prefer to use the message logging policy as the service callout policy cannot be used in the PostClientFlow. Any ideas on how I can get the content of the message recognized and resolved in Loggly using the message logging policy?
Hi @Rob Swartjes - Yes, message logging policy is convenient logging.
Can you please elaborate "message is not recognized and resolved in Loggly"? Do you see your messages in Loggly when you use Message logging policy? Also is it possible to share your message logging policy?
Hi, thanks for your swift response.
Yes I see the message in loggly when I use message logging policy. I have attached a pdf describing the issue I experience.message-logging-in-apigee.pdf
Thanks for the pdf doc.
I don't know if content type is set to JSON when you use a Message logging policy, hence you might not be seeing the values under Field Variables tab. Although I could not fix the issue for you, if put an escape "\" character while constructing the message, you get the complete message as is on Loggly.
Eg -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-1"> <DisplayName>Message Logging-1</DisplayName> <Syslog> <Message>[YOUR_TOKEN0@41058 tag="{organization.name}.{apiproxy.name}.{environment.name}"] \{ "name": "John", "age": "23", "location": "Chicago" } </Message> <Host>logs-01.loggly.com</Host> <Port>514</Port> </Syslog> </MessageLogging>
I set up the source already(Wrote Source name and tag name) in Loggly. I configured the message logging policy in apigee. Now, when I hit the API, I am checking for logs in Loggly. But can't see any.
@sudheendra1: The problem seems to be a missing version after the priority. (The <14> prefix should be <14>1). Do you know if it is possible to add the version into the syslog RFC5424 header in the message logging policy? I believe this is not correctly formatted within the message logging policy in Apigee. see link: https://tools.ietf.org/html/rfc5424#section-6.2.1 on how the syslog header should be parameterized. This issue is also referred to in other threads: https://community.apigee.com/questions/9484/configure-loggly-with-multiple-tags.html
@Peter Johnson - suggestions?
What do you see in trace? Do you see an error? Is it possible to share your message logging policy?
@sudheendra do we need to do any setup installation in loggly as i followed the above steps but i am unable to see the logs in loggly. PFA for reference.
Hi,
I have a question about the 1st approach.
Can you explain how the link between the logs Apigee is sending and your Loggly org is made?
Because I'm trying to do this and I don't see anything coming in loggly!
Thank's in advance!
Can we do this with Splunk Also ... Can anyone provide the details
Please do not share this :
https://community.apigee.com/articles/13298/log-messages-into-splunk.html
As this link explains no details of TCP Way .
I am unable to find the host details .
And some articles are showing not possible to integrate Cloud Apigee Edge with Cloud Splunk.