Proxy Logging with 3rd party service (ex. Datadog)

Background:

It is not uncommon to require the capture of logs that may only be recorded within a proxy. To meet this requirement many customers chose to use a logging service such as Datadog, Splunk, Sumo Logic, Loggly etc in Apigee. There are multiple ways to implement such services and this post will discuss and review a logging implementation specifically centered around Datadog; and hopefully answer any questions related to this question.

Implementation:

Datadog has created a great walkthrough which utilizes the Javascript policy to capture flow variables and send them via HTTP. This method differs from when using the MessageLogging Policy for a SysLog connection, which may also be possible with Datadog by using their Custom Log Forwarding via TCP.

This post will explore implementation of each of these methods.

Using a Javascript Policy (HTTP):

When using the Javascript Policy to execute the HTTP connection for sending logs, there are some disadvantages:

  1. Javascript Policies cannot execute in the PostClientFlow
  2. Javascript is an overhead cost of the proxy because it requires a new thread for execution
  3. Using the context object via context.getVariable can produce unexpected records as the context object has a global scope
  4. User created javascript can be difficult to debug and has certain limitations and it is easy to create an antipattern

From these constraints, there are multiple recommendations:

  1. Only execute this policy once within the proxy
  2. Use the AssignMessage Policy to create custom variables in case those targeted change throughout the proxy context
  3. Don't call waitForComplete()

Using a ServiceCallout Policy (HTTP):

Much like the JS Policy, the ServiceCallout Policy utilizes an HTTP connection to send logs, but it can be implemented and used in a slightly different way:

  1. The ServiceCallout policy can be used in the PostClientFlow

From these constraints, there are multiple recommendations:

  1. Again, it is recommended to use the AssignMessage policy before constructing the params/payload for the SC policy

Sample#1: Using ServiceCallout Policy for logging information to DataDog

1. Create the PostClientFlow

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
    <PreFlow name="PreFlow">
        <Request/>
        <Response/>
    </PreFlow>
    <Flows/>
    <PostFlow name="PostFlow">
        <Request/>
        <Response/>
    </PostFlow>
    <PostClientFlow>
        <Request/>
        <Response>
            <Step>
                <Name>Service-Callout-Datadog</Name>
            </Step>
        </Response>
    </PostClientFlow>
    <HTTPProxyConnection>
        <BasePath>/logging_test</BasePath>
        <VirtualHost>secure</VirtualHost>
    </HTTPProxyConnection>
    <RouteRule name="default">
        <TargetEndpoint>default</TargetEndpoint>
    </RouteRule>
</ProxyEndpoint>

2. Create the ServiceCallout policy taking the information about the request from the Datadog Javascript policy guide

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ServiceCallout async="false" continueOnError="false" enabled="true" name="Service-Callout-Datadog">
    <DisplayName>Service Callout-Datadog</DisplayName>
    <Properties/>
    <Request>
        <Set>
            <Headers>
                <Header name="Accept">application/json</Header>
            </Headers>
            <Verb>POST</Verb>
            <Payload contentType="application/json">{"message":"log from service callout"}</Payload>
        </Set>
        <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
    </Request>
    <HTTPTargetConnection>
        <Properties/>
        <URL>https://http-intake.logs.datadoghq.com/v1/input/<DATADOG API KEY>?ddsource=apigee</URL>
    </HTTPTargetConnection>
</ServiceCallout>

Using a MessageLogging Policy (TCP/UDP):

The MessageLogging Policy has multiple advantages and disadvantages compared to the JS Policy method above:

  1. The ML Policy can be used in the PostClientFlow which means that it will have a reduced impact on traffic performance that the client may experience
  2. Using this policy outside of the PostClientFlow can have performance impacts
  3. This policy is specifically for SysLog connections with TCP/UDP

From these constraints, please be advised:

  1. The log must be formatted into a message beforehand such as with an Assign Message policy or a message template.

Here is an example implementation using the custom log forwarding and MessageLogging policy as recommended:

Sample#2: Using MessageLogging Policy for logging information to Datadog

1. Create the PostClientFlow with a MessageLogging policy.

2. Create the MessageLogging Policy itself. Be sure to include your API key (host info specified here) and set PayloadOnly to be true.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-Datadog">
    <DisplayName>Message Logging-Datadog</DisplayName>
    <Syslog>
        <Message>{DATADOG API KEY} {message_to_be_logged}</Message>
        <Host>intake.logs.datadoghq.com</Host>
        <Port>10514</Port>
        <Protocol>TCP</Protocol>
        <PayloadOnly>true</PayloadOnly>
    </Syslog>
</MessageLogging>

Conclusion

Overall, there are multiple ways to implement a logging service such as Datadog. Specifically, you may use a Javascript Policy within the PreFlow or PostFlow, a ServiceCallout (recommended in the PostClientFlow), or the MessageLogging Policy. Each method has its own advantages and disadvantages; it is recommended to use the ServiceCallout or MessageLogging Policy in the PostClientFlow for the best performance.

Comments
dchiesa1
Staff

Phil, excellent stuff. Thanks for this article, really clear and helpful.

Sai-1919
Community Visitor

Hi @pindzolap , @dchiesa1 
I tried using MessageLogging Policy for logging information to Datadog but my logs are not getting inserted.
The code that I am using is below .

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-Datadog">
    <DisplayName>Message Logging-Datadog</DisplayName>
    <Syslog>
        <Message>{API KEY} {"host": "apigee", "service": "test"}</Message>
        <Host>intake.logs.datadoghq.com</Host>
        <Port>10516</Port>
        <Protocol>TCP</Protocol>
        <PayloadOnly>true</PayloadOnly>
    </Syslog>
</MessageLogging>

Thank you.

pindzolap
Staff

Hi,

In my testing (as of today, 7/7/2021), the info here is incomplete as the (2) settings below must be set:

1 - The <Port> still must be 10514

2 - The <PayloadOnly> must be set to true

Version history
Last update:
‎08-10-2021 09:11 AM
Updated by: