prerequisites for pushing APIGEE Logs to ELK

[aparnauk01]

We are trying to push APIGEE Logs to ELK . We have tried using Message Logging POLicy below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="setMessageLogging">
<DisplayName>setMessageLogging</DisplayName>
<Syslog>
<!-- add logstash IP or host withhout HTTP scheme-->
<Host>{LOGSTASH_IP_OR_HOST}</Host>
<!-- make sure the JSON message is whitespaced free, workaround to parse JSON properly at logstash side-->
<Message variablePrefix="%" variableSuffix="#">{"systemTimeStamp":"%system.time.year#-%system.time.month#-%system.time.day#T%system.time.hour#:%system.time.minute#:%system.time.second#.%system.time.millisecond#Z","messageId":"%messageid#","statusCode":"%message.status.code#","contentType":"%request.header.Content-Type#","contentLength":"%client.received.content.length#","uri":"%request.uri#","path":"%request.path#","verb":"%request.verb#","meta":{"organization":"%organization.name#","product":"%apigee.apiproduct.name#","application":"%apigee.developer.app.name#","env ":"%environment.name#","virtualhost":"%virtualhost.name#"},"proxy":{"name":"%apiproxy.name#","revision":"%apiproxy.revision#","basepath":"%proxy.basepath#","suffix":"%proxy.pathsuffix#","clientIP":"%client.ip#"},"target":{"host":"%target.host#","basepath":"%request.uri#","statusCode":"%target.response.status.code#","contentLength":"%target.received.content.length#","targetURL":"%request.url#"},"error":{"isError":"%is.error#","isPolicyError":"%apigee.edge.execution.is_policy_error#","isTargetError":"%apigee.edge.execution.is_target_error#","policyErrorPolicyName":"%apigee.edge.execution.fault_policy_name#","policyErrorFlowName":"%apigee.edge.execution.fault_flow_name#","error":"%error#","content":"%error.content#","message":"%error.message#","statusCode":"%error.status.code#","reasonPhrase":"%error.reason.phrase#","transportMessage":"%error.transport.message#","state":"%error.state#"}}</Message>
<!-- add your logstash TCP port (default 5044) -->
<Port>{LOGSTASH_PORT}</Port>
<Protocol>TCP</Protocol>
<!-- for two-way SSL, add SSLInfo, for more info - https://docs.apigee.com/api-platform/reference/policies/message-logging-policy#syslog-over-tlsssl -->
</Syslog>
<IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
<logLevel>INFO</logLevel>
</MessageLogging>

But ELK folks are asking for filebeats.yml file.

Do we really need to install Filebeats in our local ?

What are the other prerequesites from ELK side, before using message logging POlicy?

Could you please help us with steps for pushing logs to ELK.

[imesh]

As I understand Filebeats is an agent that ELK uses to publish logs to Elasticsearch:

https://www.elastic.co/guide/en/elastic-stack-get-started/7.1/get-started-elastic-stack.html#install...

If you are referring an Apigee Edge Cloud solution it would not be possible to install any log publishing agents on Edge Cloud for this purpose. Hence, we may either need to use Message Logging policy for publishing logs via TCP to Logstash using Logstash's syslog plugin:

https://docs.apigee.com/api-platform/reference/policies/message-logging-policy

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html

Or we could use a Service Callout policy (or a JavaScript policy, a Java callout, etc) for publishing logs via HTTP to Logstash's HTTP endpoint:

https://docs.apigee.com/api-platform/reference/policies/service-callout-policy

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html

If this is a Private Cloud/OPDK deployment you should be able to install an ELK log publishing agent such as Filebeat in Message Processors and other components for publishing logs to Elasticsearch:

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html