Missing syslog messages - logstash parsing
Incident Report for Swisscom Application Cloud
Identified
Dear customers

A syslog encoding change has been detected and affects the logstash parsing.
This change kept hidden while our deep testing on changing the underlaying logging architecture within the cloudfoundry update on 25th May 2020 to version 2.12.x. It counts for the recent version 2.13.x (22th June 2020) also.

Before the Platform update an underline "_" in the names (org/space) was removed before filtering on logstash.
Before the Platform update a space " " in the names (org/space) was changed to a dash "-" before filtering on logstash.
Other cases could be different, but points to similarities.

Due this different behaviour the logstash filter may fail parsing the HOSTNAME correctly.


Workaround:
Please replace this pattern on the logstash filter if you hit the "loss" of the HOSTNAME (Please check on Kibana, parsed/unparsed):
+(?:%{HOSTNAME:syslog5424_host}|-)
with:
+(?:%{DATA:syslog5424_host}|-)

This pattern change is already included in the documentation of the logstash filter as described here:
`https://docs.developer.swisscom.com/service-offerings/logstash-docker.html#create-centralized-pipeline`


Meanwhile a new release for the responsible component does resolve this issue and will be used in one of the next upcoming Platform updates. "Sanitize syslog hostnames to comply with old behavior: underscores and other invalid characters will be removed".


We apologize for the inconvenience and thanks for your understanding.

Best regards
Your Application Cloud Team
Posted Jun 26, 2020 - 15:13 CEST
This incident affects: AppCloud - Services (Elasticsearch).