Event Platform/EventGate

EventGate is an event stream intake service. It takes JSON events over HTTP POST, validates them with JSONSchema, and then produces them to a backend. The default backend (and the one used at WMF) is Kafka. More information about the EventGate NodeJS codebase can be found in its README. EventGate is meant to be generic and non WMF-specific. It can be used standalone as a library, or it can be used with the provided Express (and mediawiki-service-template) based Node HTTP server. WMF uses the service with NodeJS HTTP.

This page documents the WMF implementation and deployments of EventGate. Technical administration docs can be found at Event Platform/EventGate/Administration

EventGate code is primarily hosted on Github for greater exposure to non-Wikimedia developers. There exists plenty of tooling around using Kafka with Avro but not much for Kafka with JSONSchemas. By hosting on Github we hope to gain more visibility and participation from non-WMF developers.

WMF EventGate implementation

The WMF EventGate implementation implements validate and produce suitable for our use in WMF production. This includes configuration to look up JSON Schemas from either a the local filesystem (for eventgate-main) or a remote schema registry URL (for eventgate-analytics).

EventGate expects that specific implementations know how to map from an individual event to its JSONSchema. We use the $schema field in each of our JSON events to do this. This field contains a relative and versioned URI to the event's JSONSchema. EventGate fetches and caches this schema and uses it to validate each event with the same $schema.

Producer Types: Guaranteed and Hasty

WMF EventGates use two different Kafka producer connections, named 'guaranteed' and a 'hasty'.

The guaranteed producer is configured for low latency batching, but will block the HTTP POST request until the the event validates and the Kafka brokers ACK (or fail to ACK) the produce request for the event. Note that 'guaranteed' does not mean that the event is guaranteed to be persisted in Kafka, but that the HTTP response status can be trusted. I.e. a 2xx response guarantees that the event has been persisted.

The hasty producer is configured for high throughput batching, and will not block the HTTP POST request. Instead, it will return a 202 response as soon as EventGate has accepted the HTTP response body. The event will be validated and produced after the HTTP response is finished. An unsuccessful validate or produce for an event will be logged, but the HTTP client will not have any synchronous feedback about this.

The guaranteed producer type is the default for the /v1/events endpoint. To POST an event in hasty mode, set hasty=true in the request query parameters.

EventGate Services

At WMF EventGate is deployed to multiple service clusters, each one with a different purpose. (As of 2019-06, the services are eventgate-analytics and eventgate-main.)

EventGate services can be produced to from any production client. Mediawiki itself produces to EventGate using the EventBus extension. (Apologies if this is confusing! In writing, 'EventBus' should now refer to the Mediawiki extension, while the decommissioned eventlogging python based event service should be referred to as 'eventlogging-service-eventbus'. Event* for a disambiguation page.)

eventgate-analytics

eventgate-analytics produces events to the Kafka jumbo cluster, and is intended for high volume but low priority events. Events produced to eventgate-analytics should not be required for functional production services. The Kafka jumbo cluster only exists in eqiad, and does not have a cross-DC codfw counterpart. Events originating in codfw are produced directly to Kafka jumbo-eqiad.

eventgate-analytics (or perhaps the TBD eventgate-analytics-public) intended to replace EventLogging Analytics. It is also used for Mediawiki API and CirrusSearch request logging.

eventgate-main

eventgate-main is intended to replace eventlogging-service-eventbus. It produces events to the Kafka main clusters in both eqiad and codfw.

Development in Mediawiki Vagrant

Installation of an EventGate service in Mediawiki vagrant is optionally included via role::eventbus. To enable it, edit puppet/hieradata/common.yaml and add:

npm::node_version: 10
role::eventbus::enable_eventgate: true

EventGate requires NodeJS >= 10, but other NodeJS Mediawiki services are stuck on NodeJS 6 (for now). Ensuring that Node 10 is installed is required but may cause other NodeJS services to break.

Once done, you can enable the role and provision vagrant:

$ vagrant roles enable eventbus
$ vagrant provision

This will configure Mediawiki with the EventBus extension producing events to EventGate. The eventgate-wikimedia code repository will be cloned and the service will be launched from it using service-runner and a puppet installed config.vagrant.yaml. If you want to run the service manually (instead of letting systemd do it), you may stop it and run it with:

 nodejs /vagrant/srv/eventgate-wikimedia/node_modules/eventgate/server.js -c /vagrant/srv/eventgate-wikimedia/config.vagrant.yaml

You might want to do this if you are developing EventGate code and want to run the process in the foreground.

The eventgate-wikimedia repository contains the WMF specifc implementation of EventGate. It specifies the eventgate package as an npm depdenency. If you want to modify the behavior of the eventgate library/service, then you will need to either clone https://github.com/wikimedia/eventgate and edit, or edit node_modules/eventgate.