You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

User:Majavah/Loki notes

From Wikitech-static
Jump to navigation Jump to search

This page collects some draft notes from investigating and testing Grafana Loki as a potential solution for Toolforge log aggregation

Collection

  • promtail sidecar, with an emptyDir to log to
    • TESTED (toolsbeta.test3): this works fine at least with uwsgi, interface will be "send your logs to this file"
  • no formatting requirements! just ship your plain text
  • custom docker image with promtail and a script to create the full config file with relevant labels (at least container name, possibly deployment name as well), credentials mounted from a k8s secret
  • in a production deployment this would probably have a kubernetes mutating webhook to mount the sidecar based on a k8s annotation

Ingestion

Authentication

  • loki does not have native authentication built-in, relies on some proxy for that (but is multi-tenant, and expects the proxy to set a header)
  • promtail (official ingestion agent) supports http basic auth / client tls
  • we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret

Storage

  • loki has code for swift, but does not list as supported in official docs?
  • no support for scaling local file storage unlike OpenSearch
  • need to estimate storage needs

Querying

  • grafana is standard
    • grafana is natively multi-tenant (organizations), and data sources are per org
    • supports proxy auth but no account autocreation / support for reading groups from header
  • Grafana + CAS + custom script to create orgs? can possibly re-use parts of the script used to create grafana.wikimedia.org users from ldap

Other