You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Difference between revisions of "Event Platform/Stream Configuration"

From Wikitech-static
Jump to navigation Jump to search
imported>Ottomata
 
imported>Ottomata
Line 8: Line 8:
* The MediaWiki EventLogging extension uses stream config to vary things like event stream sampling rate.
* The MediaWiki EventLogging extension uses stream config to vary things like event stream sampling rate.
* The Analytics Cluster uses stream config to automate ingestion of streams into Hive.
* The Analytics Cluster uses stream config to automate ingestion of streams into Hive.
* EventStreams uses stream config to discover streams and auto-generate OpenAPI docs.


== Usage ==
== Usage ==
See the [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventStreamConfig/#mediawiki-config EventStreamConfig README].
See the [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventStreamConfig/#mediawiki-config EventStreamConfig README].

Revision as of 21:53, 2 December 2020

Stream configuration refers to configuration that distributed producers or consumers of a stream might want, e.g. the sampling rate or the schema title of the events that are allowed in the stream. Stream configuration was originally a requested feature of Event Platform for Product engineers, so they could more easily vary some event stream producer setting without having to do code deploys. It has since become a critical part of Event Platform, used by multiple services.

EventStreamConfig

EventStreamConfig is a MediaWiki extension that implements PHP and HTTP API for requesting stream configuration. Streams configuration entries are declared in the $wgEventStreams global list in mediawiki-config.

This centralized EventStreamConfig is used by several services to automate discovery and configuration of stream producer and consumer clients:

  • EventGate service clusters uses stream config to restrict which types of events are allowed in which streams via tha schema_title setting.
  • The MediaWiki EventLogging extension uses stream config to vary things like event stream sampling rate.
  • The Analytics Cluster uses stream config to automate ingestion of streams into Hive.
  • EventStreams uses stream config to discover streams and auto-generate OpenAPI docs.

Usage

See the EventStreamConfig README.