You are browsing a read-only backup copy of Wikitech. The live site can be found at

Logstash/Extended Retention

From Wikitech-static
< Logstash
Revision as of 01:08, 19 November 2021 by imported>Cwhite (initial draft of extended retention process)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Wikimedia infrastructure



See Privacy Policy and Data Retention Guidelines.

Request Process

  1. Ensure the log stream you want to preserve has only non-personal information not associated with a user account.
  2. Create a request to increase retention for a log stream on Phabricator, tagging the Observability team.
  3. Once the audit is complete and approved, the log stream will be tagged for inclusion into long-term retention indexes according to cluster capacity.

Audit Process

  1. Audit the log stream to determine it is non-personal information not associated with a user account.
  2. If in doubt, consult with the Security and/or Legal teams. Notify these teams when necessary.

Common Logging Schema fields indicating PII

These fields (as of ECS 1.7.0) have been identified as likely containing personal information.

  • client.*
  • error.message
  • geo.*
  • http.request.body.content
  • http.response.body.content
  • http.headers.*
  • labels
  • log.original
  • message
  • source.*
  • tls.client.*
  • user.*
  • user_agent.*