How to Monitor apigee logs with Elasticsearch and logstash
Are you using Apigee in the cloud or on-prem ? What kind of logs are you asking about ? i.e. system logs or audit logs ?
Niraj, can you please provide more info when you post the question. This will help people understand your thought process and provide you a good answers much faster.
Hi Sriki77, It was more of question rather than problem. So i had no more info to post. Let me know if you need more info on this.
Best Answer
Documentation for setting this up.
This is a great answer. If anyone is on prem and doesnt want to use syslog - i have a bunch of lessons learned about logging correctly out of the components and out of policies that im willing to share.
Answer by ozanseymen
·
Jun 22, 2015 at 08:28 AM
@Niraj - the variable that you need to use is response.status.code. So in message logging policy, Message tag, include this in curly brackets (as you are in interested in the value of the variable):
<MessageLogging name="LogToSyslog">>
<Syslog>
<Message>{response.status.code}...</Message>
Here is the full documentation of message logging policy: http://apigee.com/docs/api-services/reference/message-logging-policy
Obviously in order for this to work, you will need to put this policy in response flow - otherwise response variables will not be available for you.
All other variables that you might want to use is explained here: http://apigee.com/docs/api-services/reference/variables-reference. Just reference them in curly brackets and you are good to go.
@Ozan Seymen Thanks for the wonderful answer. I really helped me a lot to gather all the variables information at a single page. But i was unable to collect the response code as mentioned by you. I am using a single policy for both request and response. Is that something supported or we need to configure two different policy for the same.
Hi @Niraj - can you elaborate on what you mean by "unable to collect the response code" please? Are you getting an error from Apigee or you are unable to see correct data?
It would also help if you can paste your message logging policy and the resulting data you are seeing on your syslog servers.
@Ozan Seymen I don't see the status code in my kibana dashboard. There is no data been populated and the message field is blank.
you could write logs form Apigee to a syslog destination using the MessageLogging policy
http://apigee.com/docs/api-services/reference/message-logging-policy
So logstash already supports syslog as input,
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html
so it should be fairly straightforward to configure syslog input in logstash and write logs from Apigee
Thanks,
Hi Mukundha,
I am basically new to apigee and kinda learning how the MessageLogging policy works. I have created a simple yahoo weather API Proxy and trying to forward the logs to our syslog server. The syslog server i created is working fine as i have a rsyslog client that can send data to it and i can view them in kibana. (ELK setup).
Below is the message policy i am using.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <MessageLogging async="false" continueOnError="false" enabled="true" name="rsyslog"> <DisplayName>rsyslog</DisplayName> <Syslog> <Message>Message.id = {request.header.id}</Message> <Host>xx.xx.xx.xx</Host> <Port>5544</Port> <Protocol>TCP</Protocol> </Syslog> </MessageLogging>
But i don't seem to find any logs been received and seen in Kibana.
isn't syslog UDP?
plus, also check if firewall rules has been set to open port 5544 [either tcp or udp, based on your config]
It supports both UDP and TCP. And the example in Apigee Message policy documentation also describes it.
I am actually doing this right now w/ an internal set of systems. I cant promise a lot of useful data out of this - bu when i am finished i will publish some cleansed logstash configs and some brief instructions if you wish. I will NOT cover building an ELK stack - just how to get some basic and decent info out of the existing logs.
Will that help?
@Benjamin Goldman Hey !! Today i finally was able to receive the messages properly to my logstash and the same been shipped to kibana to view the dashboard data. Do let me know the setup you anywhere i can help you around.
@Benjamin Goldman @Niraj It would be awesome if you could share your learning with the community. Might save a ton of time to the next person trying to solve the same problem. Seems like a great material deserving a dedicated thread :-)
@Birute Awasthi
i will document the same and share across the community soon. :)
@Birute Awasthi @Benjamin Goldman
Attached is the documentation for the same.
