Sending Apigee Service Logs to a Remote Syslog Destination

[paulmibus]

An Apigee Private Cloud installation consists of many network services working together. Each of those services generates its own set of log files with information about configuration and runtime state. Those log files are critical for understanding cluster behavior as well as troubleshooting service errors and misconfigurations. A common question from Private Cloud customers is, "How do I get my service logs to a remote syslog destination?" Centralizing the log files and passing them through a log processing application makes the troubleshooting process much easier and can reduce the duration of an outage.

Most of the services in an Apigee cluster use either Log4J or Logback as their logging mechanism. There are some exceptions, such as OpenLDAP which logs to syslog and PostgreSQL which logs to the filesystem based on the configuration in postgresql.conf. Since changing to a remote syslog destination can involve a number of different changes for each service and may not always work well with our configuration overlay system, it is simpler and more reliable to write logs to the local filesystem and forward them on to the remote syslog destination using rsyslog, the default syslog service installed on most modern Linux distributions. Rsyslog offers the "imfile" module which monitors files and reads their contents. Once rsyslog has read those files, you can forward their contents on to any output target supported by rsyslog, including a remote destination. In addition, rsyslog provides output flexibility by offering options such as TCP delivery and TLS encryption.

Writing an rsyslog configuration file is straightforward. You can append directives to /etc/rsyslog.conf or drop a .conf file into /etc/rsyslog.d. In our case we will be using the latter method because there are many log files involved in an Apigee installation and we will want to keep those directives separate from the main system logging configuration. See the rsyslog documentation at http://www.rsyslog.com/ for full details.

Let's take a look at the configuration directives we need to forward messages to a remote syslog target. You can follow along with the documentation at http://www.rsyslog.com/doc/v7-stable/configuration/modules/imfile.html.

The first directive is a standard syslog selector:

local0.notice @syslog.local:514

This selector will send anything with a facility of "local0" and a severity of "notice" to a syslog server at the address "syslog.local" on UDP port 514. Your settings will certainly differ in at least the hostname, so check with your log aggregation server administrator for the correct settings including protocol, hostname and port. The default of local0.notice will be correct for most installations, as the imfile module sends messages there by default. If you want to change that, be aware that the input settings will need to be changed as well.

Next, we load the imfile module:

module(load="imfile" PollingInterval="10")

The module directive loads a module and optionally configures its settings. In the case of imfile the only option is PollingInterval, which defaults to 10 seconds.

Finally, we have one or more input directives that determine which files to use as inputs:

input(type="imfile"
      File="/opt/apigee/var/log/apigee-cassandra/apigee-cassandra.log"
      ReadMode="2"
      Facility="local0"
      Severity="notice"
      Tag="apigee-cassandra_apigee-cassandr"
      StateFile="apigee-cassandra_apigee-cassandra.log")

As you can see, the Facility and Severity arguments match the selector we specified above. ReadMode is set to 2, which looks for new log entries to start at character 0 in a line. That means that entries with leading whitespace such as stack traces will be interpreted as a single entry along with previous lines. The tag is an arbitrary label that is truncated because it cannot exceed 32 characters. Finally, the StateFile is used to track changes to the input file across restarts.

Once these entries are written to a file such as /etc/rsyslog.d/apigee.conf and you have restarted rsyslog (systemctl restart rsyslog), you should see messages arriving at your remote syslog host. If you don't, check your local system logs for rsyslog configuration errors or use tcpdump to verify that messages are departing.

To make this process easier, I wrote a script named apigee-rsyslog.py (attached as apigee-rsyslog.tar.gz; extract to run) that will probe a host for Apigee services and generate an rsyslog configuration file as described above. The only required argument is the address of your remote syslog host:

> python apigee-rsyslog.py syslog.local

Run the script with the -h switch to see additional options, such as facility and severity customization. A simple way to use this script would be as follows:

> python apigee-rsyslog.py syslog.local | sudo tee /etc/rsyslog.d/apigee.conf
> systemctl restart rsyslog

The script will run on Python 2.7+ and should run on Python 2.6 if you manually install the argparse module.