View source for Logstash

{{Navigation Wikimedia infrastructure|expand=logging}}
{{Ptag|Wikimedia-Logstash}}
'''Logstash''' is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

=={{anchor|Overview ("ELK+")}} Overview==
[[File:ELK_Tech_Talk_2015-08-20.pdf|thumb|Slides from TechTalk on ELK by Bryan Davis]]
[[File:Wikipedia webrequest flow 2020.png|thumb|Wikipedia request flow]]
[[File:Using_Kibana4_to_read_logs_at_Wikimedia_Tech_Talk_2016-11-14.pdf|thumb|Slides from TechTalk on Kibana4 by Bryan Davis]]

Various Wikimedia applications send log events to '''[[Logstash]]''', which gathers the messages, converts them into JSON documents, and stores them in an '''[[Elasticsearch]]''' cluster. Wikimedia uses '''Kibana''' as a front-end client to filter and display messages from the Elasticsearch cluster. These are the core components of our '''ELK stack''', but we use additional components as well. Since we utilize more than the core ELK components, we refer to our stack as "'''ELK+''''".

===Elasticsearch===

[http://www.elasticsearch.org/ Elasticsearch] is a multi-node [https://lucene.apache.org/ Lucene] implementation. The same technology powers the [[mw:Extension:CirrusSearch|CirrusSearch]] on WMF wikis.

===Logstash===

[http://logstash.net/ Logstash] is a tool to collect, process, and forward events and log messages. Collection is accomplished via configurable input plugins including raw socket/packet communication, file tailing, and several message bus clients. Once an input plugin has collected data it can be processed by any number of filters which modify and annotate the event data. Finally logstash routes events to output plugins which can forward the events to a variety of external programs including Elasticsearch, local files and several message bus implementations.

===Kibana===

[http://www.elasticsearch.org/overview/kibana/ Kibana] is a browser-based analytics and search interface for Elasticsearch that was developed primarily to view Logstash event data.

=== Kafka ===
[https://kafka.apache.org/intro Apache Kafka] is a distributed steaming system.  In our ELK stack Kafka buffers the stream of log messages produced by rsyslog (on behalf of applications) for consumption by Logstash.  Nothing should output logs to logstash directly, logs should always be sent by way of Kafka.

=== Rsyslog ===
[https://www.rsyslog.com Rsyslog] is the "rocket-fast system for log processing".  In our ELK stack rsyslog is used as the host "log agent".  Rsyslog ingests log messages in various formats and from varying protocols, normalizes them and outputs to Kafka.

=== Elastalert ===
[https://elastalert.readthedocs.io/en/latest/elastalert.html#overview Elastalert] is a utility to query log message data in order to generate alerts.  Many conditions, thresholds and output mechanisms are supported.

==Systems feeding into logstash==
See 2015-08 Tech talk slides

Writing new filters is easy.

=== Supported log shipping protocols & formats ("interfaces") ===
'''''Support of logs shipped directly from application to Logstash has been deprecated'''''.

Please see [[Logstash/Interface]] for details regarding long-term supported log shipping interfaces.

==== Kubernetes ====

Kubernetes hosted services are taken care of directly by the kubernetes infrastructure which ships via rsyslog into the logstash pipeline. All a kubernetes service needs to do is log in a JSON structured format (e.g. bunyan for nodejs services) to standard output/standard error.

===Systems not feeding into logstash===

*[[mw:Extension:EventLogging|EventLogging]] (of program-defined events with schemas), despite its name, uses a different pipeline.
*[[Varnish]] logs of the billions of[[Analytics/Pageviews | pageviews]] of WMF wikis would require a lot more hardware. Instead we use [[Kafka]] to feed [[Analytics/Data Lake/Traffic/Webrequest|web requests]] into [[Hadoop]]. A notable exception to this rule: varnish user-facing errors (HTTP status 500-599) are sent to logstash to make debugging easier.
*MediaWiki logs usually go to both logstash and log files, but a few log channels aren't. You can check which in <code>$wmgMonologChannels</code> in [https://noc.wikimedia.org/conf/InitialiseSettings.php.txt InitialiseSettings.php].

=== Writing & testing filters ===

When in the process of writing new logstash filters, take a look at what's [https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/files/logstash/ existing already] in puppet. Each filter must be tested to avoid regressions, we are using [https://github.com/magnusbaeck/logstash-filter-verifier logstash filter verifier] and existing tests can be found in the <tt>tests/</tt> directory. To write tests or run existing tests you will need logstash-filter-verifier and logstash installed locally.

Each filter has a corresponding test after its name in <tt>tests/</tt>. Within the test file the <tt>fields</tt> map lists the fields common to all tests and are used to trigger a specific filter's "if" conditions. The <tt>ignore</tt> key usually contains only <tt>@timestamp</tt> since that field is bound to change across invocations and can be safely ignored. The remainder of a test file is a list of testcases in the form of input/expected pairs. For "input" it is recommended to use yaml <tt>&gt;</tt> to include verbatim JSON, whereas "expected" is usually yaml, although it can be also verbatim JSON if more convenient.

==Production Logstash Architecture==

As of FY2019 Logstash infrastructure is owned by SRE. See also [[Logstash/SRE_onboard]] for more information on how to migrate services/applications.

===== Architecture Diagram =====
<br />
[[File:Logging Pipeline Arch Diag.jpg|center]]
<br />

===== Web interface =====

:[https://logstash.wikimedia.org logstash.wikimedia.org] runs Kibana

===== Authentication =====

:wikitech LDAP username and password and membership in one of the following LDAP groups: nda, ops, wmf

===== Configuration =====

:The cluster contains two types of nodes, configured by puppet.
:*role::logstash manages the Logstash "collector" instances.  These run logstash, a no-data Elasticsearch node, and an Apache vhost serving the Kibana application. The Apache vhosts also act as reverse proxies to the Elasticsearch cluster and perform LDAP-based authentication to restrict access to the potentially sensitive log information.
:*role::logstash::elasticsearch manages the Elasticsearch data nodes, and the Kafka-logging brokers.  These provide the inbound message buffering, and long-term storage layer for log data.

===== Hostname Convention =====

====== Current ======

:logstash1NNN Logstash related servers in [[Eqiad]].
:logstash2NNN Logstash related servers in Codfw.

====== Future ======

:logstashNNNN - Logstash "collector" hosts
:elasticsearch-loggingNNNN - Logstash Elasticsearch hosts
:kafka-loggingNNNN - Logstash Kafka broker hosts

===== Operating Systems =====
All hosts run Debian Stretch as a base operating system

===== Load Balancing and TLS =====
The misc Varnish cluster is being used to provide ssl termination and load balancing support for the Kibana application.

==Kibana quick intro==

*Start from one of the blue Dashboard links near the top, more are available from the Load icon near the top right.
*In "Events over time" click to zoom out to see what you want, or select a region with the mouse to zoom in.
**smaller time intervals are faster
**be careful you may see no events at all... because you're viewing the future
*When you get lost, click the Home icon near the top right
*As an example query, <code>wfDebugLog( <nowiki>'</nowiki>''Flow''<nowiki>'</nowiki>, ...)</code> in MediaWiki PHP corresponds to <code>type:mediawiki AND channel:flow</code>
**switch to using [[mw:Structured logging]] and you can query for ... <code>AND level:ERROR</code>

Read [{{fullurl:commons:File:ELK_Tech_Talk_2015-08-20.pdf|page=11}} slide 11 and onwards in the TechTalk on ELK by Bryan Davis], they highlight features of the Kibana web page.

==API==
The [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html Elasticsearch API] is accessible at https://logstash.wikimedia.org/elasticsearch/

Note: The [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html _search] endpoint can only be used '''without''' a request body (see {{Phabricator|T174960}}). Use [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html _msearch] instead for complex queries that need a request body.

===Extract data from Logstash with Python===
To get the last 100 log entries matching the Lucene query '''logger_name:varnishfetcherr AND layer:backend'''

<syntaxhighlight lang="python">
#!/usr/bin/env python3
import os
import sys
import json
import requests

query = "logger_name:varnishfetcherr AND layer:backend"
results = 100

ldap_user = os.getenv("LDAP_USER")
ldap_pass = os.getenv("LDAP_PASS")

if ldap_user is None or ldap_pass is None:
    print("You need to set LDAP_USER and LDAP_PASS")
    sys.exit(1)

url = "https://logstash.wikimedia.org/elasticsearch/_search?size={}&q={}".format(
    results, query
)

resp = requests.get(url, auth=requests.auth.HTTPBasicAuth(ldap_user, ldap_pass))
if resp.status_code != 200:
    print("Something's wrong, response status code={}".format(resp.status_code))
    sys.exit(1)

data = resp.json()
for line in data["hits"]["hits"]:
    print(json.dumps(line["_source"]))
</syntaxhighlight>

'''Note:''' Certain queries with whitespace characters may require additional url-encoding (via <code>urllib.parse.quote</code> or similar) when using <code>python requests</code>.  If requests to the logstash API consistently return 504 http status codes, even for relatively lightweight queries, this may be the issue.

=== Extract data from Logstash (elasticsearch) with curl and jq ===
<syntaxhighlight lang="bash">
logstash-server:~$ cat search.sh
curl -XGET 'localhost:9200/_search?pretty&size=10000' -d '
{
    "query": {
        "query_string" : {
            "query" : "facility:19,local3 AND host:csw2-esams AND @timestamp:[2019-08-04T03:00 TO 2019-08-04T03:15] NOT program:mgd"
        }
    },
    "sort": ["@timestamp"]
} '
logstash-server:~$ bash search.sh | jq '.hits.hits[]._source | {timestamp,host,level,message}' | head -20
{
  "timestamp": "2019-08-04T03:00:00+00:00",
  "host": "csw2-esams",
  "level": "INFO",
  "message": " %-: (root) CMD (newsyslog)"
}
{
  "timestamp": "2019-08-04T03:00:00+00:00",
  "host": "csw2-esams",
  "level": "INFO",
  "message": " %-: (root) CMD (   /usr/libexec/atrun)"
}
{
  "timestamp": "2019-08-04T03:01:00+00:00",
  "host": "csw2-esams",
  "level": "INFO",
  "message": " %-: (root) CMD (adjkerntz -a)"
}
$ bash search.sh | jq -r '.hits.hits[]._source | {timestamp,host,level,program,message} | map(.) | @csv' > asw2-d2-eqiad-crash.csv
</syntaxhighlight><br />
==Plugins==
We maintain a WMF specific process to build and distribute Logstash plugins to servers.

The build script and plugin git repository is located at https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/logstash/plugins
===Plugin build process===

=====Pre-requisites:=====

*An up to date Debian stable host with the Logstash package installed
*Membership to LDAP group cn=archiva-deployers,ou=groups,dc=wikimedia,dc=org

=====Build Process=====
First, create a ~/.m2/settings.xml file containing the below (see [[Archiva#Deploy to Archiva]] for additional details)<syntaxhighlight lang="xml">
# ~/.m2/settings.xml
<settings>
    <servers>
        <server>
            <id>wikimedia.releases</id>
            <username>$USERNAME</username>
            <password>XXXXXXXX</password>
        </server>
    </servers>
</settings>
</syntaxhighlight>Then, check out https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/logstash/plugins<syntaxhighlight lang="bash">
git clone https://gerrit.wikimedia.org/r/operations/software/logstash/plugins
</syntaxhighlight>In the newly created directory 'plugins' , initialize git-fat<syntaxhighlight lang="bash">
cd plugins
git-fat init
</syntaxhighlight>Then execute the build script, providing the logstash version for which plugins should be built as the first argument<syntaxhighlight lang="bash">
./build.sh 5.6.15    # 5.6.15 is used as an example, substitute in desired logstash version
</syntaxhighlight>Or, if rebuilding plugins at the same version, for example to add a new plugin, you may append a local version number like so<syntaxhighlight lang="bash">
# To optionally rebuild plugins for the same logstash version

./build.sh 5.6.15-1   # 5.6.15 local version 1
</syntaxhighlight>


Also git add and git commit the changes. Then send it for review<syntaxhighlight lang="bash">
git add .
git commit -m "Upgrade logstash plugin to 5.X.X"
git review -R

</syntaxhighlight>After review and merging the change, you'll need to pull the change onto the deployment server, and run a scap deploy.

Currently the deploy server is deploy1001<syntaxhighlight lang="bash">
# ssh to deploy master, currently deploy1001

cd /srv/deployment/logstash/plugins/

# have a look at what is in the local working copy. you'll want to bring this up to date
# with the gerrit repository but also not inadvertantly clobber manual changes

git pull

# if you wish to see what would happen without actually running the deploy
scap deploy --dry-run --no-log-message

# deploy the updated plugins
scap deploy "updating logstash plugins to $version"
</syntaxhighlight>After a successful scap deploy, puppet should do the right thing upon the next agent run to fetch and install the update plugins

Puppet will not restart logstash.  This must be done manually in a rolling fashion, and it's strongly suggested to perform this in step with the plugin deploy. 
<br />
{{anchor|Prototype (Beta) Logstash}}
==Beta Cluster Logstash==

;Web interface
:[https://logstash-beta.wmflabs.org/ logstash-beta.wmflabs.org]
;Configuration
:It hosts a functional Logstash + Elasticsearch + Kibana stack at [https://logstash-beta.wmflabs.org/ logstash-beta.wmflabs.org] that aggregates log data produced by the [[Nova_Resource:Deployment-prep|beta cluster]]. It is hosted on the VM [[toolforge:openstack-browser/server/deployment-logstash2.deployment-prep.eqiad.wmflabs|deployment-logstash2.deployment-prep.eqiad.wmflabs]].
;Access control
:Unlike production Logstash, Beta Cluster may not use LDAP for authentication. Credentials for Beta's Logstash can be found on <code>deployment-deploy01.deployment-prep.eqiad.wmflabs</code> in <code>/root/secrets.txt</code>.

==Gotchas==
===GELF transport===
Make sure logging events sent to the GELF input don't have a "type" or "_type" field set, or if set, that it contains the value "gelf". The gelf/logstash config discards any events that have a different value set for "type" or "_type". The final "type" seen in Kibana/Elasticsearch will be take from the "facility" element of the original GELF packet. The application sending the log data to Logstash should set "facility" to a reasonably unique value that identifies your application.

==Documents==
{{Special:Prefixindex/Logstash/|hideredirects=1|stripprefix=1}}

== Troubleshooting ==

=== Kafka consumer lag ===

For a host of reasons it might happen that there's a buildup of messages on Kafka. For example:
; elasticsearch is refusing to index messages, thus logstash can't consume properly from kafka.
: The reason for index failure is usually conflicting fields, see also {{bug|T150106}} for a detailed discussion of the problem. The solution is to find what programs are generating the conflicts and drop them on logstash accordingly, see also {{bug|T228089}}

== Operations ==

=== Configuration changes ===

Puppet will restart logstash as needed upon configuration changes (supporting reload is TODO), after merging your configuration change it is usually enough to run cumin in batches of one with >60s of sleep:

  cumin -b1 -s60 'O:logstash or O:logstash7' 'run-puppet-agent -q'

==== Test a configuration snippet before merge ====
Copy your ready to merge snippet (eg. modules/profile/files/logstash/filter-syslog-network.conf) to a Logstash host.
Then run 
  sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f <myfile>

It should return "Configuration OK".

=== Indexing errors ===

We're alerting on errors that Logstash gets from Elasticsearch whenever there's an "indexing conflict" between fields of the same index (see also {{Bug|T236343}}). The reason usually is because two applications send logs with the same field name but two different types, e.g. <code>response</code> will be sent as a string in one case but as nested object in another. {{Bug|T239458}} is a good example of this, where different parts of mediawiki send logs formatted in a different way.

=== No logs indexed ===

This alert is based on the incoming logs per second indexed by Elasticsearch. During normal operation there is a baseline of ~1k logs/s (July 2020) and anything significantly lower than that is an unexpected condition. Check the Logstash dashboard attached to alert for signs of root causes. Most likely Logstash has stopped sending logs to Elasticsearch.

== Stats ==

=== Documents and bytes counts ===

The Elasticsearch ''cat'' API provides a simple way to extract general statistics about log storage, e.g. total logs and bytes (not including replication)

  logstash1010:~$ curl -s 'localhost:9200/_cat/indices?v&bytes=b' | awk '/logstash-/ {b+=$10; d+=$7} END {print d; print b}'

Or logs per day (change $3 to $7 to get bytes sans replication)

  logstash1010:~$ curl -s 'localhost:9200/_cat/indices?v&bytes=b' | awk '/logstash-/ { gsub(/logstash-[^0-9]*/, "", $3); sum[$3] += $7 } END { for (i in sum) print i, sum[i] }' | sort

Or logs per month:

  logstash1010:~$ curl -s 'localhost:9200/_cat/indices?v&bytes=b' | awk '/logstash-/ { gsub(/logstash-[^0-9]*/, "", $3); gsub(/\.[0-9][0-9]$/, "", $3); sum[$3] += $7 } END { for (i in sum) print i, sum[i] }' | sort

==See also==

*[[mw:Manual:Structured logging]] (MediaWiki part of the job to feed into Logstash)
*[[Logs#mw-log]] (the old method of viewing logs)
*[[phabricator:J177|Introducing Phatality]] (a Kibana plugin to streamline the process of reporting production errors on Phabricator)

[[Category:Services]]