Message logging - Syslog TLS/SSL - Issue with log message with line breaks

Prashant819
Participant II

Hello,

I have a query on message logging policy when it is used with UDP and TCP/SSL to send log message with line breaks to Splunk.

Currently we are in the process of migrating from Apigee on-prem to Apigee Cloud and have a on-prem Splunk setup.

1. With on-prem Apigee --> When syslog over UDP is used to send a log message (with line breaks) to Splunk . It is sent as a single message and appear in Splunk as a single message.

Message logging policy configuration -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="true" enabled="true" name="ML-CommonLogMessage">
    <Syslog>
        <Message>{logMessage}</Message>
        <Host>SplunkHostName</Host>
        <Port>SplunkUDPPort</Port>
    </Syslog>
</MessageLogging>

2. With cloud Apigee --> When syslog with TCP/SSL is used to send a log message (with line breaks) to Splunk. It gets split into multiple messages on each line break and appear in splunk as multiple messages.

I want this to be a single message in this case too (same as syslog/UDP from OPDK).

Message logging policy configuration -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="true" enabled="true" name="CommonLogMessage">
    <Syslog>
        <Message>{logMessage}</Message>
        <Host>SplunkHostName</Host>
        <Port>SplunkTCPPort</Port>
        <Protocol>TCP</Protocol>
        <SSLInfo>
            <Enabled>true</Enabled>
            <ClientAuthEnabled>true</ClientAuthEnabled>
            <KeyStore>ref://Ref_Keystore</KeyStore>
            <KeyAlias>Alias_Keystore</KeyAlias>
            <TrustStore>ref://Ref_Truststore</TrustStore>
        </SSLInfo>
    </Syslog>
</MessageLogging>

My question is - Why Apigee sends log message with line breaks as multiple messages (and not as a single message) with TCP/SSL?

Please suggest if I am missing something or my understanding is wrong.

@Dino-at-Google

Thanks.

1 4 1,071
4 REPLIES 4

Hi Prashant

Are you saying that {logMessage} has line breaks?

And when that occurs, then you get multiple messages in Splunk, and that's not what you want. Is that right??

Maybe the way to avoid this problem is to eliminate the line breaks in the log Message?

Maybe try using the replaceAll static function in a AssignMessage ?

<AssignMessage name='AM-CleanLinebreaks'>
  <AssignVariable>
    <Name>space</Name>
    <Value> </Value>
  </AssignVariable>
  <AssignVariable>
    <Name>regex</Name>
    <Value>\n</Value>
  </AssignVariable>
  <AssignVariable>
    <Name>modifiedData</Name>
    <Template>{replaceAll(originalData,regex,space)}</Template>
  </AssignVariable>
</AssignMessage>

Hello Dino,

Yes, {logmessage} variable has few name value pairs including request.content and response.content context variables.

In case request or response json/xml payload has linefeeds, I see that each line from payload is received as a new message on rsyslog server.

However this was works fine if UDP is used to send the same logmesaage. Complete logmessage including payload is received as a single message.

I prefer not modifying all proxies with workaround of removing linefeeds before logging if there is something else to be fixed

Thanks.

Hi @Dino-at-Google,

I am facing the same issue. Any reason as to why every new line goes as a separate message to the splunk server?

If you have a choice to use logstash - then you can use multiline codec to have messages with newline chars retained as a single message/event without splitting.

For this it is preferred to enable FormatMessage to true in the MessageLogging policy which gives you a fixed string to patternize.

For egs: add the below codec to the syslog input section of your logstash configuration

codec =>  multiline {
          # Grok pattern names are valid! :)
          pattern => "^<[0-9]*>[0-9]*\s%{TIMESTAMP_ISO8601}\s.+\sApigee\s-\s"
          negate => true
          what => "previous"
        }

Please note the pattern may need tweaking egs: For Apigee hybrid we see the text Apigee-Edge - - - instead of Apigee - - -. . In the above example we have used a negate =true - which means it will match all lines not having the pattern text at the beginning of the line. 'what' will cause all such lines to be combined with preceding lines which will roll up to the pattern text at the beginning of the message. For more help - you can refer logstash references.

In our case we were using apigee hybrid and we could see that the message is not getting split from apigee side but at the receiving side - in our case logstash. So using the multiline codec in the above manner we were able to proceed without having the single message (with newlines) not getting split into multiple messages