You are browsing a read-only backup copy of Wikitech. The primary site can be found at


From Wikitech-static
< Logstash
Revision as of 19:30, 31 May 2019 by imported>Cwhite
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Supported Interfaces

Systemd Journal (stdout/stderr)

Structured logs written to stdout/stderr will be picked up by journald and copied to rsyslog. From there, rsyslog decides whether or not to forward the log to Kafka and eventually to Logstash.

To indicate to rsyslog that the log message is JSON, a "cookie" is required. Prepending "@cee: " before the JSON blob is sufficient.[1]

NOTE: Logs messages are broken between lines if they are longer than 2048 characters The fix in systemd is available in Debian Buster.

Python implementation

import logging
import logging.config
from pythonjsonlogger import jsonlogger

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        log_record['level'] = record.levelname.upper()

class StructuredLoggingHandler(logging.StreamHandler):
    def __init__(self, rsyslog=False):
        super(StructuredLoggingHandler, self).__init__()
        if rsyslog:
            prefix = '@cee: '
            prefix = ''
        self.formatter = CustomJsonFormatter(prefix=prefix)

# Demo code below
if __name__ == '__main__':
        'version': 1,
        'disable_existing_loggers': False,
        'root': {
            'handlers': ['demo'],
            'level': 'DEBUG'
        'handlers': {
            'demo': {
                'class': 'logger_demo.StructuredLoggingHandler',
                'rsyslog': True

    })'It\'s log!')

UNIX Socket (/dev/log)


Tailing Log Files


Configuring rsyslog to forward your logs

Rsyslog needs to know that your logs should be forwarded to Kafka. There are two configuration items that must be in place.

Your application may need to set the SyslogIdentifier option under the Service heading in the systemd unit file. This is especially true for applications that run under a common runtime like Python or Java.


The application must also be listed in the rsyslog lookup table and configured to flag the log for sending to Kafka.