You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
EventGate is an event stream intake service. It takes JSON events over HTTP POST, validates them with JSONSchema, and then produces them to a backend. The default backend (and the one used at WMF) is Kafka. More information about the EventGate NodeJS codebase can be found in its README. EventGate is meant to be generic and non WMF-specific. It can be used standalone as a library, or it can be used with the provided Express (and mediawiki-service-template) based Node HTTP server. WMF uses the service with NodeJS HTTP.
This page documents the WMF implementation and deployments of EventGate. Technical administration docs can be found at Event Platform/EventGate/Administration
EventGate code is primarily hosted on Github for greater exposure to non-Wikimedia developers. There exists plenty of tooling around using Kafka with Avro but not much for Kafka with JSONSchemas. By hosting on Github we hope to gain more visibility and participation from non-WMF developers.
WMF EventGate implementation
The WMF EventGate implementation implements validate and produce suitable for our use in WMF production. This includes configuration to look up JSON Schemas from either a the local filesystem (for eventgate-main) or a remote schema registry URL (for eventgate-analytics).
EventGate expects that specific implementations know how to map from an individual event to its JSONSchema. We use the $schema field in each of our JSON events to do this. This field contains a relative and versioned URI to the event's JSONSchema. EventGate fetches and caches this schema and uses it to validate each event with the same $schema.
Producer Types: Guaranteed and Hasty
WMF EventGates use two different Kafka producer connections, named 'guaranteed' and a 'hasty'.
The guaranteed producer is configured for low latency batching, but will block the HTTP POST request until the the event validates and the Kafka brokers ACK (or fail to ACK) the produce request for the event. Note that 'guaranteed' does not mean that the event is guaranteed to be persisted in Kafka, but that the HTTP response status can be trusted. I.e. a 2xx response guarantees that the event has been persisted.
The hasty producer is configured for high throughput batching, and will not block the HTTP POST request. Instead, it will return a 202 response as soon as EventGate has accepted the HTTP response body. The event will be validated and produced after the HTTP response is finished. An unsuccessful validate or produce for an event will be logged, but the HTTP client will not have any synchronous feedback about this.
The guaranteed producer type is the default for the /v1/events endpoint. To POST an event in hasty mode, set hasty=true in the request query parameters.
At WMF EventGate is deployed to multiple service clusters, each one with a different purpose. (As of 2020-02, the services are eventgate-main, eventgate-analytics, eventgate-analytics-external, and eventgate-logging-external.)
EventGate services can be produced to from any production client. Mediawiki itself produces to EventGate using the EventBus extension. (Apologies if this is confusing! In writing, 'EventBus' should now refer to the Mediawiki extension, while the decommissioned eventlogging python based event service should be referred to as 'eventlogging-service-eventbus'. See Event* for a disambiguation page.)
eventgate-main produces events to the Kafka main clusters in both eqiad and codfw. It is used for low(ish) volume but high priority events. These events are necessary for functioning of Wikimedia core services, like the Mediawiki Job Queue and change-propagation.
- EventBus MediaWiki extension. (Config)
- restbase (resource_change events for purges)
eventgate-analytics produces events to the Kafka jumbo cluster, and is intended for high volume but low priority events. Events produced to eventgate-analytics should not be required for functional production services. The Kafka jumbo cluster only exists in eqiad, and does not have a cross-DC codfw counterpart. Events originating in codfw are produced directly to Kafka jumbo-eqiad.
- EventBus MediaWiki extension. (Config)
- WDQS Query Logging. (Config)
- analytics/refinery Oozie util/swift/upload workflow. (Config) and Discovery/Search's glent oozie jobs (as of 2020-02).
- Discovery/Search's airflow swift upload plugin.
eventgate-analytics-external replaces EventLogging Analytics.
It uses both dynamic stream configuration (via the EventStreamConfig extension MW API) dynamic schemas (via https://schema.wikimedia.org) to validate events from external clients (like the EventLogging extension) that produce to https://intake-analytics.wikimedia.org/v1/events. eventgate-analytics-external produces events to the Kafka jumbo cluster.
- WikimediaEvents (via EventLogging)
- Popups Extension (This should be moved to use EventLogging)
eventgate-logging-external accepts mediawiki/client/error events from external clients at https://intake-logging.wikimedia.org/v1/events. eventgate-logging produces events to the Kafka logging cluster.
All EventGate Services are configured to send validation error events to Kafka. These events are ingested into Hive into the various
event.eventgate_*_error_validation tables, but are also ingested into logstash and viewable in Kibana using this saved search.
Development in Mediawiki Vagrant
Installation of an EventGate service in Mediawiki vagrant is included via role::eventbus. To install it, first edit puppet/hieradata/common.yaml and add:
EventGate requires NodeJS >= 10, but other NodeJS Mediawiki services are stuck on NodeJS 6 (for now). Ensuring that Node 10 is installed is required but may cause other NodeJS services to break.
Once done, you can enable the role and provision vagrant:
$ vagrant roles enable eventbus $ vagrant provision
This will configure Mediawiki with the EventBus extension producing events to EventGate. The eventgate-wikimedia code repository will be cloned and the service will be launched from it using service-runner and a puppet installed config.vagrant.yaml. If you want to run the service manually (instead of letting systemd do it), you may stop it and run it with:
nodejs /vagrant/srv/eventgate-wikimedia/node_modules/eventgate/server.js -c /vagrant/srv/eventgate-wikimedia/config.vagrant.yaml
You might want to do this if you are developing EventGate code and want to run the process in the foreground.
The eventgate-wikimedia repository contains the WMF specifc implementation of EventGate. It specifies the eventgate package as an npm depdenency. If you want to modify the behavior of the eventgate library/service, then you will need to either clone https://github.com/wikimedia/eventgate and edit, or edit node_modules/eventgate.