{ Community }
  • Academy
  • Docs
  • Developers
  • Resources
    • Community Articles
    • Apigee on GitHub
    • Code Samples
    • Videos & eBooks
    • Accelerator Methodology
  • Support
  • Ask a Question
  • Spaces
    • Product Announcements
    • General
    • Edge/API Management
    • Developer Portal (Drupal-based)
    • Developer Portal (Integrated)
    • API Design
    • APIM on Istio
    • Extensions
    • Business of APIs
    • Academy/Certification
    • Adapter for Envoy
    • Analytics
    • Events
    • Hybrid
    • Integration (AWS, PCF, Etc.)
    • Microgateway
    • Monetization
    • Private Cloud Deployment
    • 日本語コミュニティ
    • Insights
    • IoT Apigee Link
    • BaaS/Usergrid
    • BaaS Transition/Migration
    • Apigee-127
    • New Customers
    • Topics
    • Questions
    • Articles
    • Ideas
    • Leaderboard
    • Badges
  • Log in
  • Sign up

Get answers, ideas, and support from the Apigee Community

  • Home /
  • Edge/API Management /
avatar image
9

Log messages into Splunk  

  • Export to PDF
sarthak created · Nov 20, 2015 at 05:41 AM · 19.6k Views · Dino edited · Feb 28, 2017 at 05:37 PM

Logging into an external logging system is a very common requirement for EDGE platform and one of the most common systems we need to log to is splunk.

There are multiple ways of logging into Splunk. Below are described a few ways to log:

For trying out this I installed Splunk enterprise on-prem.

1. Log over HTTP

As a step 1 we first configure HTTP Event Collector in Splunk. I did that following the tutorial here:

http://dev.splunk.com/view/event-collector/SP-CAAAE7F

Once you finish this then Splunk will be listening on port 8088 by default for incoming requests over HTTP/s

Then create a Service callout policy with a configuration like below to send logs over HTTP/s event Collector.

<ServiceCallout name="Log-to-Splunk">
    <DisplayName>Log to Splunk</DisplayName>
    <Properties/>
    <Request clearPayload="true" variable="myRequest">
        <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
        <Set>
            <Headers>
                <Header name="Authorization">Splunk PUT-SPLUNK-CODE_HERE</Header>
            </Headers>    
            <Payload contentType="application/json">
                {"event":"Logged from EDGE"}
            </Payload>    
            <Path>services/collector/event</Path>
            <Verb>POST</Verb>
        </Set>
    </Request>
    <Response>calloutResponse</Response>
    <HTTPTargetConnection>
        <Properties/>
        <URL>https://52.23.227.40:8088/</URL>
    </HTTPTargetConnection>
</ServiceCallout>

The logs appear like below in Splunk:

2. Log over TCP

The other option is to log via TCP or UDP using Message Logging Policy.

Both Apigee and splunk supports both the protocols. But Splunk recommends TCP, hence we will use that.

The policy will look like this:

<MessageLogging name="Log-to-splunk-over-TCP">
    <DisplayName>Log to splunk over TCP</DisplayName>
    <Syslog>
        <Message>Message = "Something happened and I am logging via TCP"</Message>
        <Host>XX.XX.XX.XX</Host>
        <Port>2900</Port>
        <Protocol>TCP</Protocol>
    <SSLInfo>
        <Enabled>true</Enabled>
    </SSLInfo>
    </Syslog>
</MessageLogging>

3. Log via javascript

You will also be able to log to splunk via javascript or node.js. You can find corresponding tutorials here: http://dev.splunk.com/view/javascript-sdk/SP-CAAAEC9

4. As @gbhandari points out there is one more option for logging into Splunk. But this is only applicable for Apigee on-prem (private cloud) customers.

You can use the message logging policy to log messages to local file system.Check here: http://docs.apigee.com/api-services/reference/message-logging-policy#location. You can install Splunk agents on those instances. Each Splunk agent can forward the messages logged to the Splunk server.

search-splunk-6-3-1.png (24.3 kB)
thub.nodes.view.add-new-comment
loggingsplunkdevops
Add comment Show 12
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Alex Koo ♦♦   · Mar 29, 2016 at 11:26 PM 1
Link

+1'd. And if async JS calls are the soup du jour, great post on doing that in Apigee Edge here: https://community.apigee.com/articles/2340/asynchronous-http-requests-in-an-api-proxy.html

avatar image gbhandari   · Mar 30, 2016 at 10:47 AM 0
Link

@sarthak

Hi Sarthak,

There is one more way.

One can also install splunk on all the servers. And in the splunk configuration files, you can provide the important log details like system-monitor.logs, system.logs, Message Logging Policy logs, etc log paths and those log files will be forwarded to splunk.

avatar image sarthak ♦ gbhandari · Apr 01, 2016 at 05:24 PM 0
Link

Hi @gbhandari Yes, thanks that is a good suggestion. But it is only applicable for on-prem customers. I will update my article with this.

avatar image SuganyaSamuel · Apr 04, 2016 at 09:30 PM 0
Link

If SSLInfo = enabled in messaging policy, how to specify which trustore to use in this Policy ?

avatar image Dino ♦♦ SuganyaSamuel   · Feb 28, 2017 at 05:40 PM 0
Link

You would do that in the same way you configure TrustStore for any SSL-enabled target in Apigee Edge.

<SSLInfo> 
      <Enabled>true</Enabled> 
      <TrustStore>myTrustStore</TrustStore>
</SSLInfo> 

See here for the documentation

avatar image Anto Shojan Dino ♦♦ · Jan 23, 2019 at 05:47 AM 0
Link

Log to splunk over TCP Message = "Something happened and I am logging via TCP" xxx.xxx.com xxxx TCP true true xxxxx ref://xxxkeystoreref ref://xxxtruststoreref TLSv1.2

Show more comments
avatar image Dino ♦♦   · Feb 28, 2017 at 05:42 PM 0
Link

The benefit of using JavaScript callout via the httpClient is that you can make it fire-and-forget. You do not need to wait for the response. See here for a relevant Q&A.

avatar image Kristopher Kleva · Feb 28, 2017 at 05:45 PM 1
Link

MessageLogging policies appear to perform the best in my testing so it's the approach I've recommended here a few times. Especially if you want to be logging multiple times in a request (e.g. Request, Response and any Faults) .

I would not recommend doing it in JavaScript or Node.js unless you do not have any other alternative. I've tried this myself and run into performance issues at high loads.

Finally, don't forget about the PostClientFlow for logging timings after the response has been delivered to the client.

    <PostClientFlow>
      <Request/>
      <Response>
          <Step>
              <Name>Message-Logging-Response</Name>
          </Step>
      </Response>
    </PostClientFlow>
avatar image Arun Prasath Sivanesan · Dec 17, 2018 at 03:48 PM 0
Link

while logging over TCP, where can i find the value of <Host> tag. FYI: i am using trail version of both splunk and apigee

avatar image Avdhesh Chouhan Arun Prasath Sivanesan · Apr 15, 2020 at 05:40 PM 0
Link

I am facing the same issue . From last 4 hours I am trying to locate host name in splunk for TCP ... It should be easy to find.

avatar image Anto Shojan · Jan 23, 2019 at 05:50 AM 0
Link
Log to splunk over TCP Message = "Something happened and I am logging via TCP" xxx.xxx.com xxxx TCP true true xxxxxxx ref://xxxxkeystoreref ref://xxxtruststoreref TLSv1.2

Article

Contributors

avatar image avatar image avatar image

Follow this article

32 People are following this .

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Navigation

Log messages into Splunk

Related Articles

3 new capabilities available in Apigee Edge (September)

Log Messages into Loggly

Logging to Sumo Logic using JavaScript and HTTP

Debugging, logging, monitoring and performance tuning of Edge node.js proxies

Stackdriver Monitoring and Logging For Edge

Apigee Edge - External Message Logging to ELK stack

Using scheduling for Apigee trace instead of manual work with control

Basic Apigee CICD Pipeline - updated

  • Products
    • Edge - APIs
    • Insights - Big Data
    • Plans
  • Developers
    • Overview
    • Documentation
  • Resources
    • Overview
    • Blog
    • Apigee Institute
    • Academy
    • Documentation
  • Company
    • Overview
    • Press
    • Customers
    • Partners
    • Team
    • Events
    • Careers
    • Contact Us
  • Support
    • Support Overview
    • Documentation
    • Status
    • Edge Support Portal
    • Privacy Policy
    • Terms & Conditions
© 2021 Apigee Corp. All rights reserved. - Apigee Community Terms of Use - Powered by AnswerHub
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Post an idea
  • Spaces
  • Product Announcements
  • General
  • Edge/API Management
  • Developer Portal (Drupal-based)
  • Developer Portal (Integrated)
  • API Design
  • APIM on Istio
  • Extensions
  • Business of APIs
  • Academy/Certification
  • Adapter for Envoy
  • Analytics
  • Events
  • Hybrid
  • Integration (AWS, PCF, Etc.)
  • Microgateway
  • Monetization
  • Private Cloud Deployment
  • 日本語コミュニティ
  • Insights
  • IoT Apigee Link
  • BaaS/Usergrid
  • BaaS Transition/Migration
  • Apigee-127
  • New Customers
  • Explore
  • Topics
  • Questions
  • Articles
  • Ideas
  • Badges