Log Messages into Loggly

One of the best ways to track down problems in APIs is through logging. The easiest way to do that In Apigee Edge is to attach a Message Logging policy on your API and log custom messages to a local disk or to syslog. When configured for syslog, it will forward log messages from Apigee Edge to a remote syslog server including external log management services, such a Splunk, Sumo Logic, and Loggly.

This article covers some of the common methods to log into Loggly from Apigee Edge. If you are interested in logging to Splunk, then jump to this article.

Pre-requisites

You should have an account with Loggly. I am using my free trial account for creating this article. In this post I will discuss two approaches to build this integration. However both the approaches require you to have an unique Consumer Token. You can get your Consumer Key from the Loggly management UI, under Source Setup tab.

1687-loggly-1.jpg

Approach 1 : Using Message Logging Policy

You can attach a Message Logging policy either on the request or response path of your API. A simple Message Logging policy will look like this -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-1">
    <DisplayName>Message Logging-1</DisplayName>
    <Syslog>
        <Message>[YOUR_LOGGLY_CONSUMER_TOKEN@41058 tag="{organization.name}.{apiproxy.name}.{environment.name}"] MESSAGE_YOU_WANT_TO_LOG </Message>
        <Host>logs-01.loggly.com</Host>
        <Port>514</Port>
    </Syslog>
</MessageLogging>

Let's take a closer look at this policy, especially the section under <Syslog>.

<Message> : Represents the syslog message to be logged. This includes the header, tag (s) and the actual content. Loggly only accepts syslog events with a RFC5424 compliant header. Within the header there must be a structured data set that includes your Customer Token and the Loggly private enterprise number (PEN) which is 41058 (a constant). Tags are great for adding organization to your log events to aid in segmentation & filtering. They are metadata you can set on the source side, and can be included with any event that is transmitted to Loggly.

<Host> & <Port> : Loggly provides two syslog endpoints that accept both UDP and TCP syslog. For standard syslog please use:

 logs-01.loggly.com:514 

For secure syslog encrypted with TLS please use:

 logs-01.loggly.com:6514. 

In the Message Logging policy, these are represented under <Host> and <Port> sections.

Approach 2 : Using ServiceCallout Policy

Loggly provides standard HTTP based API interfaces for sending data. You can send a single line, multiple lines or bulk event data using these interfaces. With Apigee Edge, you can call this interface using a Service callout policy. A sample policy will look like this -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ServiceCallout async="false" continueOnError="false" enabled="true" name="Service-Callout-1">
    <DisplayName>Service Callout-1</DisplayName>
    <Properties/>
    <Request variable="myRequest">
        <Set>
            <Headers>
                <Header name="content-type">text/plain</Header>
            </Headers>
            <Payload>
                MESSAGE_YOU_WANT_TO_LOG 
            </Payload>
        </Set>
    </Request>
    <Response>calloutResponse</Response>
    <HTTPTargetConnection>
        <Properties/>
        <URL>http://logs-01.loggly.com/inputs/YOUR_LOGGLY_CONSUMER_TOKEN/tag/http/</URL>
    </HTTPTargetConnection>
</ServiceCallout> 

NOTE : While using these interfaces you don't have to append **@41058** to your Consumer Token.

Viewing Logs in Loggly

The logs in Loggly can be viewed either on the Dashboard or by searching based on the tag.

1688-loggly-2.jpg

NOTE : This article focus only on Apigee Edge and Loggly integration. Please refer to Loggly documentation if you need additional information of their capabilities.

References -

UPDATE :

We have fixed issue with Loggly message formatting while using message logging policy, for more details please refer article here.

Comments
Not applicable

I have implemented logging as described in this entry (both approach 1 and 2). However when I use the Message Logging Policy (approach1) I do not get the desired result in Loggly as with approach 2 using the service callout policy. With the message logging policy the content of the message is not recognized and resolved in Loggly as it is with the service callout policy. I prefer to use the message logging policy as the service callout policy cannot be used in the PostClientFlow. Any ideas on how I can get the content of the message recognized and resolved in Loggly using the message logging policy?

sudheendras
Staff

Hi @Rob Swartjes - Yes, message logging policy is convenient logging.

Can you please elaborate "message is not recognized and resolved in Loggly"? Do you see your messages in Loggly when you use Message logging policy? Also is it possible to share your message logging policy?

Not applicable

Hi, thanks for your swift response.

Yes I see the message in loggly when I use message logging policy. I have attached a pdf describing the issue I experience.message-logging-in-apigee.pdf

sudheendras
Staff

Thanks for the pdf doc.

I don't know if content type is set to JSON when you use a Message logging policy, hence you might not be seeing the values under Field Variables tab. Although I could not fix the issue for you, if put an escape "\" character while constructing the message, you get the complete message as is on Loggly.

Eg -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-1">
    <DisplayName>Message Logging-1</DisplayName>
    <Syslog>
        <Message>[YOUR_TOKEN0@41058 
                tag="{organization.name}.{apiproxy.name}.{environment.name}"] 
                \{ 
                  "name":  "John",
                  "age": "23",
                  "location":  "Chicago"
                }
        </Message>
        <Host>logs-01.loggly.com</Host>
        <Port>514</Port>
    </Syslog>
</MessageLogging>
Not applicable

I set up the source already(Wrote Source name and tag name) in Loggly. I configured the message logging policy in apigee. Now, when I hit the API, I am checking for logs in Loggly. But can't see any.

Not applicable

@sudheendra1: The problem seems to be a missing version after the priority. (The <14> prefix should be <14>1). Do you know if it is possible to add the version into the syslog RFC5424 header in the message logging policy? I believe this is not correctly formatted within the message logging policy in Apigee. see link: https://tools.ietf.org/html/rfc5424#section-6.2.1 on how the syslog header should be parameterized. This issue is also referred to in other threads: https://community.apigee.com/questions/9484/configure-loggly-with-multiple-tags.html

sudheendras
Staff

@Peter Johnson - suggestions?

sudheendras
Staff

What do you see in trace? Do you see an error? Is it possible to share your message logging policy?

Not applicable

@sudheendra do we need to do any setup installation in loggly as i followed the above steps but i am unable to see the logs in loggly. PFA for reference.

loggly.pngmessagelogedge.png

Not applicable

Hi,

I have a question about the 1st approach.
Can you explain how the link between the logs Apigee is sending and your Loggly org is made?

Because I'm trying to do this and I don't see anything coming in loggly!

Thank's in advance!

avdheshchouhan-
Explorer

sudheendras

Can we do this with Splunk Also ... Can anyone provide the details

Please do not share this :

https://community.apigee.com/articles/13298/log-messages-into-splunk.html

As this link explains no details of TCP Way .

I am unable to find the host details .

And some articles are showing not possible to integrate Cloud Apigee Edge with Cloud Splunk.

Version history
Last update:
‎12-18-2015 03:29 PM
Updated by: