You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

User:Majavah/Loki notes: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Majavah
No edit summary
 
imported>Majavah
No edit summary
Line 17: Line 17:
* promtail (official ingestion agent) supports http basic auth / client tls
* promtail (official ingestion agent) supports http basic auth / client tls
* we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret
* we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret
** '''TESTED''': [https://git.sr.ht/~taavi/toolforge-log-platform proof of concept] works!


== Storage ==
== Storage ==

Revision as of 20:01, 26 December 2021

This page collects some draft notes from investigating and testing Grafana Loki as a potential solution for Toolforge log aggregation

Collection

  • promtail sidecar, with an emptyDir to log to
    • TESTED (toolsbeta.test3): this works fine at least with uwsgi, interface will be "send your logs to this file"
  • no formatting requirements! just ship your plain text
  • custom docker image with promtail and a script to create the full config file with relevant labels (at least container name, possibly deployment name as well), credentials mounted from a k8s secret
  • in a production deployment this would probably have a kubernetes mutating webhook to mount the sidecar based on a k8s annotation

Ingestion

Authentication

  • loki does not have native authentication built-in, relies on some proxy for that (but is multi-tenant, and expects the proxy to set a header)
  • promtail (official ingestion agent) supports http basic auth / client tls
  • we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret

Storage

  • loki has code for swift, but does not list as supported in official docs?
  • no support for scaling local file storage unlike OpenSearch
  • need to estimate storage needs

Querying

  • grafana is standard
    • grafana is natively multi-tenant (organizations), and data sources are per org
    • supports proxy auth but no account autocreation / support for reading groups from header
  • Grafana + CAS + custom script to create orgs? can possibly re-use parts of the script used to create grafana.wikimedia.org users from ldap