You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Logstash
Logstash is a tool for managing events and logs. When used generically the term encompases a larger system of log collection, processing, storage and searching activities.
Overview
Logstash is used to gather logging messages, convert them into json documents and store them in an Elasticsearch cluster. Kibana is used as a frontend client to search for and display messages from Elasticsearch cluster.
Logstash
Logstash is a tool that can be used to collect, process and forward events and log messages. Collection is accomplished via number of configurable input plugins including raw socket/packet communication, file tailing and several message bus clients. Once an input plugin has collected data it can be processed by any number of filters which modify and annotate the event data. Finally events are routed to output plugins which can forward the events to a variety of external programs including Elasticsearch, local files and several message bus implementations.
Elasticsearch
Elasticsearch is a multi-node Lucene implementation. The same technology powers the CirrusSearch on WMF wikis.
Kibana
Kibana is a browser based analytics and search interface for Elasticsearch that was developed primarily to view Logstash event data.
Production Logstash
- Web interface
- logstash.wikimedia.org
- Authentication
- wikitech LDAP username and password and membership in the "wmf" LDAP group.
- Hosts
- logstash100[1-3] servers in Eqiad.
- Configuration
- Each host provides a Logstash instance, an Elasticsearch node, a Redis server and an Apache vhost serving the Kibana application. The Apache vhosts also act as reverse proxies to the Elasticsearch cluster and perform LDAP-based authentication to restrict access to the potentially sensitive log information. The misc Varnish cluster is being used to provide ssl termination and load balancing support.
Prototype (Beta) Logstash
- Web interface
- logstash-beta.wmflabs.org
- Authentication
- Limited access; The username and password can be found on
deployment-bastion.eqiad.wmflabs
in the/root/secrets.txt
file. - Hosts
- deployment-logstash1.eqiad.wmflabs
- Configuration
- It hosts a functional Logstash + Elasticsearch + Kibana stack at logstash-beta.wmflabs.org that aggregates log data produced by the beta cluster.
Documents
Gotchas
GELF transport
Make sure logging events sent to the GELF input don't have a "type" or "_type" field set, or if set, that it contains the value "gelf". The gelf/logstash config discards any events that have a different value set for "type" or "_type".