{ Community }
  • Academy
  • Docs
  • Developers
  • Resources
    • Community Articles
    • Apigee on GitHub
    • Code Samples
    • Videos & eBooks
    • Accelerator Methodology
  • Support
  • Ask a Question
  • Spaces
    • Product Announcements
    • General
    • Edge/API Management
    • Developer Portal (Drupal-based)
    • Developer Portal (Integrated)
    • API Design
    • APIM on Istio
    • Extensions
    • Business of APIs
    • Academy/Certification
    • Adapter for Envoy
    • Analytics
    • Events
    • Hybrid
    • Integration (AWS, PCF, Etc.)
    • Microgateway
    • Monetization
    • Private Cloud Deployment
    • 日本語コミュニティ
    • Insights
    • IoT Apigee Link
    • BaaS/Usergrid
    • BaaS Transition/Migration
    • Apigee-127
    • New Customers
    • Topics
    • Questions
    • Articles
    • Ideas
    • Leaderboard
    • Badges
  • Log in
  • Sign up

Get answers, ideas, and support from the Apigee Community

  • Home /
  • Edge/API Management /
avatar image
0
Question by Niraj · Jun 18, 2015 at 05:59 PM · 3.9k Views messagelogging

How to Monitor apigee logs with Elasticsearch and logstash

Comment
Add comment Show 3
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image sarthak ♦ · Jun 18, 2015 at 06:02 PM 0
Link

Are you using Apigee in the cloud or on-prem ? What kind of logs are you asking about ? i.e. system logs or audit logs ?

avatar image sriki77 ♦ · Jun 24, 2015 at 03:27 AM 0
Link

Niraj, can you please provide more info when you post the question. This will help people understand your thought process and provide you a good answers much faster.

avatar image Niraj sriki77 ♦ · Jun 24, 2015 at 03:49 AM 0
Link

Hi Sriki77, It was more of question rather than problem. So i had no more info to post. Let me know if you need more info on this.

Close

4 Answers

  • Sort: 
avatar image
4
Best Answer

Answer by Niraj · Jun 25, 2015 at 06:48 PM

Documentation for setting this up.


apigee-message-logging.pdf (108.8 kB)
Comment
Add comment Show 1 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Benjamin Goldman · Jul 23, 2015 at 05:43 PM 0
Link

This is a great answer. If anyone is on prem and doesnt want to use syslog - i have a bunch of lessons learned about logging correctly out of the components and out of policies that im willing to share.

avatar image
2

Answer by ozanseymen   · Jun 22, 2015 at 08:28 AM

@Niraj - the variable that you need to use is response.status.code. So in message logging policy, Message tag, include this in curly brackets (as you are in interested in the value of the variable):

<MessageLogging name="LogToSyslog">>
  <Syslog>
    <Message>{response.status.code}...</Message>

Here is the full documentation of message logging policy: http://apigee.com/docs/api-services/reference/message-logging-policy

Obviously in order for this to work, you will need to put this policy in response flow - otherwise response variables will not be available for you.

All other variables that you might want to use is explained here: http://apigee.com/docs/api-services/reference/variables-reference. Just reference them in curly brackets and you are good to go.

Comment
Add comment Show 7 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Niraj · Jun 23, 2015 at 05:27 AM 0
Link

@Ozan Seymen Thanks for the wonderful answer. I really helped me a lot to gather all the variables information at a single page. But i was unable to collect the response code as mentioned by you. I am using a single policy for both request and response. Is that something supported or we need to configure two different policy for the same.

avatar image ozanseymen ♦♦ Niraj   · Jun 23, 2015 at 07:20 AM 0
Link

Hi @Niraj - can you elaborate on what you mean by "unable to collect the response code" please? Are you getting an error from Apigee or you are unable to see correct data?

It would also help if you can paste your message logging policy and the resulting data you are seeing on your syslog servers.

avatar image Niraj ozanseymen ♦♦ · Jun 23, 2015 at 06:57 PM 0
Link

@Ozan Seymen I don't see the status code in my kibana dashboard. There is no data been populated and the message field is blank.

Show more comments
avatar image
0

Answer by gnanasekaran · Jun 19, 2015 at 02:16 AM

you could write logs form Apigee to a syslog destination using the MessageLogging policy

http://apigee.com/docs/api-services/reference/message-logging-policy

So logstash already supports syslog as input,

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html

so it should be fairly straightforward to configure syslog input in logstash and write logs from Apigee

Thanks,

Comment
Add comment Show 11 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Niraj · Jun 19, 2015 at 04:35 AM 0
Link

Hi Mukundha,

I am basically new to apigee and kinda learning how the MessageLogging policy works. I have created a simple yahoo weather API Proxy and trying to forward the logs to our syslog server. The syslog server i created is working fine as i have a rsyslog client that can send data to it and i can view them in kibana. (ELK setup).

Below is the message policy i am using.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <MessageLogging async="false" continueOnError="false" enabled="true" name="rsyslog"> <DisplayName>rsyslog</DisplayName> <Syslog> <Message>Message.id = {request.header.id}</Message> <Host>xx.xx.xx.xx</Host> <Port>5544</Port> <Protocol>TCP</Protocol> </Syslog> </MessageLogging>

But i don't seem to find any logs been received and seen in Kibana.

avatar image gnanasekaran ♦ Niraj · Jun 19, 2015 at 04:39 AM 0
Link

isn't syslog UDP?

plus, also check if firewall rules has been set to open port 5544 [either tcp or udp, based on your config]

avatar image Niraj gnanasekaran ♦ · Jun 19, 2015 at 04:41 AM 0
Link

It supports both UDP and TCP. And the example in Apigee Message policy documentation also describes it.

Show more comments
avatar image
0

Answer by Benjamin Goldman · Jun 25, 2015 at 04:26 PM

@Niraj

I am actually doing this right now w/ an internal set of systems. I cant promise a lot of useful data out of this - bu when i am finished i will publish some cleansed logstash configs and some brief instructions if you wish. I will NOT cover building an ELK stack - just how to get some basic and decent info out of the existing logs.

Will that help?

Comment
Add comment Show 4 · Link
10 |5000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by Apigeeks only
  • Viewable by the original poster
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image Niraj · Jun 25, 2015 at 05:46 PM -1
Link

@Benjamin Goldman Hey !! Today i finally was able to receive the messages properly to my logstash and the same been shipped to kibana to view the dashboard data. Do let me know the setup you anywhere i can help you around.

avatar image birute@google.com ♦♦   · Jun 25, 2015 at 06:09 PM 0
Link

@Benjamin Goldman @Niraj It would be awesome if you could share your learning with the community. Might save a ton of time to the next person trying to solve the same problem. Seems like a great material deserving a dedicated thread :-)

avatar image Niraj birute@google.com ♦♦ · Jun 25, 2015 at 06:13 PM 1
Link
@Birute Awasthi

i will document the same and share across the community soon. :)

avatar image Niraj Niraj · Jun 25, 2015 at 06:46 PM 0
Link

@Birute Awasthi @Benjamin Goldman

Attached is the documentation for the same.

Follow this Question

Answers Answers and Comments

19 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

How to log entire Proxy Request, Proxy Response, Target Request and Target Response to Third Party systems like Loggly? 2 Answers

Message Logging and Sumologic 2 Answers

MessageLogging and DataMasking 1 Answer

Setting up Logging with Sumo Logic - which Host should I use? 5 Answers

Free logging softwares for message logging? 2 Answers

  • Products
    • Edge - APIs
    • Insights - Big Data
    • Plans
  • Developers
    • Overview
    • Documentation
  • Resources
    • Overview
    • Blog
    • Apigee Institute
    • Academy
    • Documentation
  • Company
    • Overview
    • Press
    • Customers
    • Partners
    • Team
    • Events
    • Careers
    • Contact Us
  • Support
    • Support Overview
    • Documentation
    • Status
    • Edge Support Portal
    • Privacy Policy
    • Terms & Conditions
© 2021 Apigee Corp. All rights reserved. - Apigee Community Terms of Use - Powered by AnswerHub
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Create an article
  • Post an idea
  • Spaces
  • Product Announcements
  • General
  • Edge/API Management
  • Developer Portal (Drupal-based)
  • Developer Portal (Integrated)
  • API Design
  • APIM on Istio
  • Extensions
  • Business of APIs
  • Academy/Certification
  • Adapter for Envoy
  • Analytics
  • Events
  • Hybrid
  • Integration (AWS, PCF, Etc.)
  • Microgateway
  • Monetization
  • Private Cloud Deployment
  • 日本語コミュニティ
  • Insights
  • IoT Apigee Link
  • BaaS/Usergrid
  • BaaS Transition/Migration
  • Apigee-127
  • New Customers
  • Explore
  • Topics
  • Questions
  • Articles
  • Ideas
  • Badges