Logging to Sumo Logic using JavaScript and HTTP

Sumo Logic is a popular cloud based logging solution. They support local file upload, SysIog and HTTP Source collection types. For API logging out of Apigee, the HTTP endpoint is the way to go. The Syslog endpoint requires the installation of a "collector" component which isn't possible with Apigee.

This solution uses a single JavaScript policy to make HTTP POST requests to Sumo Logic HTTP Source Collector.

Using the JavaScript policy to make asynchronous HTTP requests is a standard approach documented here.

Caveat: JavaScript policies cannot be used in the PostClientFlow, but that's OK, put them in the Proxy PostFlow or wherever else you want to log, like in the DefaultFaultFlow.

Setting up logging to Sumo is really simple, here's a step by step guide with a realistic and working JavaScript policy that shows data being logged in Sumo Logic.

Setup Sumo Logic

1. Create a free trial account at Sumo Logic.

2. Select "Set Up Streaming Data" using the setup Wizard.

3572-sumo-1.png

3. Select the data type "Your Custom App"

3573-sumo-2.png

4. Set up the collection using "HTTP Source"

3574-sumo-3.png

5. Configure the HTTP source, this can be what ever hierarchy you want.

3581-sumo-4.png

6. Save the obfuscated URL to be put into your policy (or into a configuration KVM in a real world scenario), there is no security other than the URL value itself.

Grab a cup of coffee, wait for the email from Sumo indicating your log is ready (few minutes, maybe more).

Add Logging to your Proxy

In the meantime, you can configure your API to send log entries. In this example, I use the following JavaScript policy in the Proxy Post Flow:

// Get variable which I set from a previous KVM configuration lookup.
// I use this to turn logging on or off
// var logging = context.getVariable("exco.logging");
var logging = "true";

if (logging == "true") {

    // Get variable from previous KVM configuration lookup
    // var logServerURL = context.getVariable("exco.loggingUrl");
    var logServerURL = "https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV1oma90Vvb...";
    
    // Debug
    print('LOGGING ' + logServerURL);
    
    // calculate response times for client, target and total
    var request_start_time = context.getVariable('client.received.start.timestamp');
    var target_start_time  = context.getVariable('target.sent.start.timestamp');
    var target_end_time    = context.getVariable('target.received.end.timestamp');
    var request_end_time   = context.getVariable('system.timestamp');
    var total_request_time = request_end_time-request_start_time;
    var total_target_time  = target_end_time-target_start_time;
    var total_client_time  = total_request_time-total_target_time;

    // Create the log object which can be queried by field in Sumo
    var logObject = {
        "organization": context.getVariable("organization.name"),
        "environment": context.getVariable("environment.name"),
        "apiProduct": context.getVariable("apiproduct.name"),
        "proxyName": context.getVariable("apiproxy.name"),
        "appName": context.getVariable("developer.app.name"),
        "verb": context.getVariable("request.verb"),
        "url": '' + context.getVariable("client.scheme") + '://' + context.getVariable("request.header.host") + context.getVariable("request.uri"),
        "responseCode": context.getVariable("message.status.code"),
        "responseReason": context.getVariable("message.reason.phrase"),
        "clientLatency": total_client_time,
        "targetLatency": total_target_time,
        "totalLatency": total_request_time
    };


    var headers = {
        'Content-Type': 'application/json'
    };


    // Debug
    print('LOGGING OBJECT' + JSON.stringify(logObject));

    var myLoggingRequest = new Request(logServerURL, "POST", headers, JSON.stringify(logObject));


    httpClient.send(myLoggingRequest);
}

Test the Solution

Now fire off a few requests and login to Sumo to search the logs, I just searched "*", you can build more complete queries.

3583-sumo-6.png

Create Dashboards

There are many ways to structure your logs, by environments, by APIs, etc., take some time to figure out how you want to view the logs. In a large enterprise with lots of logs in Sumo Logic, I created top level for apigee, then environments (dev, test, prod), then APIs. Then I create dashboards with various views, for example:

3584-sumo-7.png

Happy Logging!

Comments
Not applicable

Great guide.

Can you further explain how does the SUMO http URL prevent getting data from other sources besides mine? (e.g. malicious usage )

kurtkanaskie
Staff

Thanks @Yair I had the same question and Sumo's answer is there is no way to prevent others from sending data to the endpoint, other than through URL obfuscation. In other words, the endpoint URL is very long and not guessable.

For example (real endpoint with one character changed):

https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/ZaVnC4dhaV0g8UJyNIL8tHwGYLAFN3upnNlq...

Docs here say to "Keep this address private since anyone can use it to send data"

Not applicable

(This is Brian Goleno, Product Manager at Sumo Logic)

The API URL is "/receiver/v1/http/"

the string at the end is a unique key, a secret, for you to manage and keep private. This key is bound to a "Source", what we call a data collection entry point to a data stream. You can associate metadata with a Source when creating it, or you can override that metadata with headers. Because this API only accepts HTTPS requests, the token is not exposed during the transactions.

Some customers take additional measures. You can have as many of these "sources" as you wish. Each one can have the same (or different) metadata. Each one will have its own unique key. You can use this for a key rotation scheme, or to shard your clients across many keys. Additionally, you can add another token/string to the body of the message, and then add a filter rule to the Source that will not accept logs that do not contain that additional string.

cheers,

Brian

edu4krishanu
Participant V
@Kurt Kanaskie

Great Article!

However I am getting this error. Any Idea?

Execution of Policy_JS_WriteLogInSumoLogic failed with error: Javascript runtime error: "Access to Java class "com.apigee.nio.util.Bytes$BytesInputStream" is prohibited. (Policy_JS_WriteLogInSumoLogic#32)"

Thanks,

Krish

davissean
Staff

Hi @Krish

Remove the print() statement and everything should be fine

Many thanks, Sean

kurtkanaskie
Staff

Thanks @Sean Davis, I recall the print() statement introduced a bug, but don't remember which release. I've not seen it happen in my usage so I'm assuming its fixed now. I'm using Cloud Edge 170118.

edu4krishanu
Participant V

I got it resolved. APIGEE was throwing error to do JSON.stringify.

davissean
Staff
DChiesa
Staff

By the way, this is a good example of sending "fire and forget" HTTP requests from JavaScript callout. It works with other systems, and using this approach for logging is a general pattern. See here for a relevant Q&A.

jeremyhill
Explorer

This is great exactly what I was looking for. Just one question. If I wanted to send to sumologic from an on prem private cloud install but have to pass through a webproxy how would you suggest I do that?

robert
Participant V

Great article @Kurt Kanaskie! Must have missed this one!

Hi @Jeremy Hill. Perhaps you should create a new question and post it to the community?

bradknowles
Explorer

So, if you put this into the Default Proxy Endpoint Postflow, it seems to only capture successful transactions. If we run into an error condition, it never gets logged. If the error is caught within Apigee itself and never gets to the target, the response code is sent to the client without ever triggering the Target Endpoint Postflow. If the error is generated by the target and returned to Apigee, the Target Response Flow is started, but it still doesn't trigger the Postflow.

Any suggestions?

kurtkanaskie
Staff

That's correct @BradKnowles, as mentioned in the "Caveat" above, you should also add the JavaScript policy to the DefaultFaultRule in your proxy endpoint.

    <DefaultFaultRule name="DefaultFaultRule">
        <Step>
            <Name>JS-Log</Name>
        </Step>
        <AlwaysEnforce>true</AlwaysEnforce>
    </DefaultFaultRule>
bradknowles
Explorer

Awesome! Now I'm also seeing errors showing up in SumoLogic.

The only question I've got left is what happens when there is a problem with the target, but the target doesn't actually return an error code, but maybe just closes the connection? In that case, the proxy will come back and report a 502 or 503 error, but will that case also be logged to SumoLogic? This is actually the very specific case we're trying to track down now -- when the target error is "null", but the proxyResponseCode is a 5xx error.

kurtkanaskie
Staff

Backend errors >= 4xx will trigger the fault flow and will eventually make it to your DefaultFaultRule in the proxy (assuming no intervening Raise Faults or other conditions).

Try changing your TargetServer to something like "foo.com" and see what happens.

Version history
Last update:
‎09-23-2016 02:45 PM
Updated by: