You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Rsyslog

From Wikitech-static
Revision as of 13:36, 9 April 2019 by imported>Filippo Giunchedi (First edition, add troubleshooting)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

rsyslog is the default Debian logging daemon and what's deployed fleet-wide at Wikimedia Foundation.

Troubleshooting

rsyslog "stuck"

rsyslog has been observed for getting stuck from time to time (e.g. its tls listener stops responding). In these situations a restart "fixes" the problem, however before doing a restart it is important to capture the daemon' status:

  cd
  timeout 30s strace -f -p $(pidof rsyslogd) -s 65535 -o rsyslog_$(date -Im).strace
  lsof -p $(pidof rsyslogd) > rsyslog_$(date -Im).lsof
  gdb -p $(pidof rsyslogd) --batch -ex gcore
  gdb -p $(pidof rsyslogd) --batch -ex 'thread apply all bt full' > rsyslog_$(date -Im).threaddump
  systemctl restart rsyslog