It is not uncommon to require the capture of logs that may only be recorded within a proxy. To meet this requirement many customers chose to use a logging service such as Datadog, Splunk, Sumo Logic, Loggly etc in Apigee. There are multiple ways to implement such services and this post will discuss and review a logging implementation specifically centered around Datadog; and hopefully answer any questions related to this question.
Datadog has created a great walkthrough which utilizes the Javascript policy to capture flow variables and send them via HTTP. This method differs from when using the MessageLogging Policy for a SysLog connection, which may also be possible with Datadog by using their Custom Log Forwarding via TCP.
This post will explore implementation of each of these methods.
When using the Javascript Policy to execute the HTTP connection for sending logs, there are some disadvantages:
From these constraints, there are multiple recommendations:
Much like the JS Policy, the ServiceCallout Policy utilizes an HTTP connection to send logs, but it can be implemented and used in a slightly different way:
From these constraints, there are multiple recommendations:
1. Create the PostClientFlow
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ProxyEndpoint name="default"> <PreFlow name="PreFlow"> <Request/> <Response/> </PreFlow> <Flows/> <PostFlow name="PostFlow"> <Request/> <Response/> </PostFlow> <PostClientFlow> <Request/> <Response> <Step> <Name>Service-Callout-Datadog</Name> </Step> </Response> </PostClientFlow> <HTTPProxyConnection> <BasePath>/logging_test</BasePath> <VirtualHost>secure</VirtualHost> </HTTPProxyConnection> <RouteRule name="default"> <TargetEndpoint>default</TargetEndpoint> </RouteRule> </ProxyEndpoint>
2. Create the ServiceCallout policy taking the information about the request from the Datadog Javascript policy guide
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ServiceCallout async="false" continueOnError="false" enabled="true" name="Service-Callout-Datadog"> <DisplayName>Service Callout-Datadog</DisplayName> <Properties/> <Request> <Set> <Headers> <Header name="Accept">application/json</Header> </Headers> <Verb>POST</Verb> <Payload contentType="application/json">{"message":"log from service callout"}</Payload> </Set> <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables> </Request> <HTTPTargetConnection> <Properties/> <URL>https://http-intake.logs.datadoghq.com/v1/input/<DATADOG API KEY>?ddsource=apigee</URL> </HTTPTargetConnection> </ServiceCallout>
The MessageLogging Policy has multiple advantages and disadvantages compared to the JS Policy method above:
From these constraints, please be advised:
Here is an example implementation using the custom log forwarding and MessageLogging policy as recommended:
1. Create the PostClientFlow with a MessageLogging policy.
2. Create the MessageLogging Policy itself. Be sure to include your API key (host info specified here) and set PayloadOnly to be true.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-Datadog"> <DisplayName>Message Logging-Datadog</DisplayName> <Syslog> <Message>{DATADOG API KEY} {message_to_be_logged}</Message> <Host>intake.logs.datadoghq.com</Host> <Port>10514</Port> <Protocol>TCP</Protocol> <PayloadOnly>true</PayloadOnly> </Syslog> </MessageLogging>
Overall, there are multiple ways to implement a logging service such as Datadog. Specifically, you may use a Javascript Policy within the PreFlow or PostFlow, a ServiceCallout (recommended in the PostClientFlow), or the MessageLogging Policy. Each method has its own advantages and disadvantages; it is recommended to use the ServiceCallout or MessageLogging Policy in the PostClientFlow for the best performance.
Phil, excellent stuff. Thanks for this article, really clear and helpful.
Hi @pindzolap , @dchiesa1
I tried using MessageLogging Policy for logging information to Datadog but my logs are not getting inserted.
The code that I am using is below .
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-Datadog"> <DisplayName>Message Logging-Datadog</DisplayName> <Syslog> <Message>{API KEY} {"host": "apigee", "service": "test"}</Message> <Host>intake.logs.datadoghq.com</Host> <Port>10516</Port> <Protocol>TCP</Protocol> <PayloadOnly>true</PayloadOnly> </Syslog> </MessageLogging>
Thank you.
Hi,
In my testing (as of today, 7/7/2021), the info here is incomplete as the (2) settings below must be set:
1 - The <Port> still must be 10514
2 - The <PayloadOnly> must be set to true