Difference between revisions of "Event Platform/EventGate"

From Wikitech-static
Jump to navigation Jump to search
imported>Ottomata
imported>Hashar
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[https://github.com/wikimedia/eventgate EventGate] is an event stream intake service.  It takes JSON events over HTTP POST, validates them with JSONSchema, and then produces them to a backend.  The default backend (and the one used at WMF) is Kafka.  More information about the EventGate NodeJS codebase can be found in its [https://github.com/wikimedia/eventgate/blob/master/README.md README]. EventGate is meant to be generic and non WMF-specific.  It can be used standalone as a library, or it can be used with the provided Express (and mediawiki-service-template) based Node HTTP server.  WMF uses the service with NodeJS HTTP.
{{Navigation Event Platform}}


This page documents the WMF implementation and deployments of EventGate.  Technical administration docs can be found at [[Event Platform/EventGate/Administration]]


'''EventGate''' is an HTTP service for ingestion of events, written in Node.js. It takes JSON messages over HTTP POST requests, optionally validates them against a JSONSchema, and then produces them to a backend.  The default backend (and the one used at WMF) is Kafka.
For more information about the EventGate codebase, see the README at [https://github.com/wikimedia/eventgate#readme github.com/wikimedia/eventgate]. EventGate is meant to be generic and not WMF specific.  It can be used standalone as a library, or it can be used with the built-in support for running [https://expressjs.com/ Express] (the Node.js HTTP server provided via mediawiki-service-template). WMF operates EventGate using the Express Node.js HTTP server.
This page documents the WMF deployments of EventGate.
== Purpose ==
Event Platform seeks to standardize the process of producing and consuming event streams at WMF.  EventGate is a an HTTP proxy for producing events that follows our practices.  Specifically, it does the following:
* Receives events of a specific schema version destined to a specific stream.
* Uses EventStreamConfig to ensure that only events of the same schema lineage are emitted to a stream.
* Augments the event with some custom defaults (e.g. setting date time fields or HTTP header data in the event).
* Looks up the schema of the incoming event, and validates that the event data conforms to the schema.
* Adds [[Kafka#Multi_Datacenter|datacenter prefixes]] to the destination stream name to make the destination Kafka topic name.
* Produces to Kafka.
{{notice|Any Event Platform producer should be able to do all of these things.}}
== Service ==
{{Main|Event Platform/EventGate/Administration}}
=== Implementation ===
{{See|See https://github.com/wikimedia/eventgate and https://gerrit.wikimedia.org/g/eventgate-wikimedia/}}
EventGate code is primarily hosted on Github for greater exposure to non-Wikimedia developers. There exists plenty of tooling around using Kafka with Avro but not much for Kafka with JSONSchemas.  By hosting on Github we hope to gain more visibility and participation from non-WMF developers.
EventGate code is primarily hosted on Github for greater exposure to non-Wikimedia developers. There exists plenty of tooling around using Kafka with Avro but not much for Kafka with JSONSchemas.  By hosting on Github we hope to gain more visibility and participation from non-WMF developers.


= WMF EventGate implementation =
The <code>eventgate-wikimedia</code> repository contains the WMF deployment of EventGateIt uses the eventgate package as an npm dependency, and has additional utilities, configuration and deployment pipeline code for WMF's instances of EventGate.
The [https://gerrit.wikimedia.org/r/plugins/gitiles/eventgate-wikimedia/ WMF EventGate implementation] implements <tt>validate</tt> and <tt>produce</tt> suitable for our use in WMF productionThis includes configuration to look up JSON Schemas from either a the local filesystem (for eventgate-main) or a remote schema registry URL (for eventgate-analytics).


EventGate expects that specific implementations know how to map from an individual event to its JSONSchema. We use the <tt>$schema</tt> field in each of our JSON events to do this.  This field contains a relative and versioned URI to the event's JSONSchema. EventGate fetches and caches this schema and uses it to validate each event with the same <tt>$schema</tt>.
To modify the behavior of the eventgate library or service, use EventGate directly instead, from https://github.com/wikimedia/eventgate.


== Producer Types: Guaranteed and Hasty ==
=== Operation ===
WMF EventGates use two different Kafka producer connections, named 'guaranteed' and a 'hasty'.
There are multiple clusters of EventGate services in Wikimedia production.


The guaranteed producer is configured for low latency batching, but will block the HTTP POST request until the the event validates and the Kafka brokers ACK (or fail to ACK) the produce request for the event. Note that 'guaranteed' does not mean that the event is guaranteed to be persisted in Kafka, but that the HTTP response status can be trusted.  I.e. a 2xx response guarantees that the event has been persisted.
EventGate is hosted in production as a containerized service running on [[Kubernetes]], as such there are no Puppet roles or persistent backend hostnames.


The hasty producer is configured for high throughput batching, and will not block the HTTP POST request. Instead, it will return a 202 response as soon as EventGate has accepted the HTTP response body. The event will be validated and produced after the HTTP response is finished.  An unsuccessful validate or produce for an event will be logged, but the HTTP client will not have any synchronous feedback about this.
As of Feb 2020, there are currently the following clusters (per [[git:operations/deployment-charts/+/HEAD/charts/eventgate|deployment-charts]]):


The guaranteed producer type is the default for the <tt>/v1/events</tt> endpoint.  To POST an event in hasty mode, set <tt>hasty=true</tt> in the request query parameters.
* eventgate-main
* eventgate-analytics
* eventgate-analytics-external
* eventgate-logging-external


== EventGate Services ==
More details on how are used further down below.
At WMF EventGate is deployed to multiple service clusters, each one with a different purpose.  (As of 2020-02, the services are eventgate-main, eventgate-analytics, eventgate-analytics-external, and eventgate-logging-external.)


EventGate services can be produced to from any production client.  Mediawiki itself produces to EventGate using the [https://www.mediawiki.org/wiki/Extension:EventBus EventBus extension](Apologies if this is confusing! In writing, 'EventBus' should now refer to the Mediawiki extension, while the decommissioned eventlogging python based event service should be referred to as 'eventlogging-service-eventbus'.  See [[Event*]] for a disambiguation page.)
==Wikimedia EventGate configuration==
{{See|See https://gerrit.wikimedia.org/g/eventgate-wikimedia/}}
Wikimedia's EventGate wrapper implements custom <code>validate</code> and <code>produce</code> behaviours for our use in WMF productionThis includes configuration to look up JSON Schemas from either a the local filesystem (for eventgate-main) or a remote schema registry URL (for eventgate-analytics).


=== eventgate-main ===
EventGate expects that specific implementations know how to map from an individual event to its JSONSchema.  We use the <code>$schema</code> field in each of our JSON events to do thisThis field contains a relative and versioned URI to the event's JSONSchema. EventGate fetches and caches this schema and uses it to validate each event with the same <code>$schema</code>.
eventgate-main produces events to the Kafka main clusters in both eqiad and codfw.  It is used for low(ish) volume but high priority events.  These events are necessary for functioning of Wikimedia core services, like the Mediawiki Job Queue and change-propagation.


==== Clients ====
===Producer types: Guaranteed and Hasty===
* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventBus/+/master EventBus MediaWiki extension].  ([https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/master/wmf-config/ProductionServices.php#45 Config])
Wikimedia's EventGate configuration offers two different Kafka producer connections, named <code>guaranteed</code> and <code>hasty</code>.
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/hieradata/role/common/scb.yaml#148 change-propagation]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/hieradata/role/common/restbase/production.yaml#21 restbase] (resource_change events for purges)


=== eventgate-analytics ===
The '''guaranteed''' producer will block the HTTP response until the event has been validated and sent to the Kafka brokers with either an ACK response or known failure to ACK. Note that "guaranteed" does not mean that the event is guaranteed to be persisted in Kafka (there is not an indefinite retry or other buffer). Rather it means that HTTP response status can be trusted, so a 2xx status code guarantees that the event has been persisted.
eventgate-analytics produces events to the Kafka jumbo cluster, and is intended for high volume but low priority events.  Events produced to eventgate-analytics should not be required for functional production services.  The Kafka jumbo cluster only exists in eqiad, and does not have a cross-DC codfw counterpart.  Events originating in codfw are produced directly to Kafka jumbo-eqiad.


==== Clients ====
The '''hasty''' producer is optimised for high-throughput, and will not block the HTTP response. Instead, it will immediately return a 202 status response as soon as EventGate has received the JSON message from the HTTP response body. The event will be validated and produced to Kafka afterwardThe HTTP client that submitted the event will not know whether the event was valid or whether it was succesfully persisted in Kafka. If the event failed validation or failed production to Kafka, an error will be logged to Logstash however.
* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventBus/+/master EventBus MediaWiki extension]([https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/master/wmf-config/ProductionServices.php#44 Config])
* WDQS Query Logging. ([https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/hieradata/role/common/wdqs.yaml#12 Config])
* analytics/refinery Oozie util/swift/upload workflow. ([https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/master/oozie/util/swift/upload/workflow.xml#57 Config]) and Discovery/Search's glent oozie jobs (as of 2020-02).
* Discovery/Search's airflow [https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/discovery/analytics/+/master/airflow/plugins/swift_upload_plugin.py swift upload plugin].


=== eventgate-analytics-external ===
The guaranteed producer type is the default for the <code>/v1/events</code> endpoint.  To POST an event in hasty mode, set <code>hasty=true</code> in the request query parameters.
eventgate-analytics-external replaces [[Analytics/Systems/EventLogging|EventLogging Analytics]].


It uses both dynamic stream configuration (via the EventStreamConfig extension MW API) dynamic schemas (via https://schema.wikimedia.org) to validate events from external clients (like the EventLogging extension) that produce to https://intake-analytics.wikimedia.org/v1/events.  eventgate-analytics-external produces events to the Kafka jumbo cluster.
==EventGate clusters==
At WMF, EventGate is deployed as multiple separate clusters, each with its own defined purpose.


==== Clients ====
All EventGate clusters are open to receive events from any internal production server.  MediaWiki produces to EventGate using the [[mw:Extension:EventBus|EventBus extension]]. (Apologies if this is confusing! See [[Event*]] for a disambiguation page for related terms.)
* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EventLogging/+/master EventLogging]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/master WikimediaEvents] (via EventLogging)
* [https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Popups/+/422261 Popups] Extension (This should be moved to use EventLogging)


=== eventgate-logging-external ===
===eventgate-main===
eventgate-logging-external accepts [https://schema.wikimedia.org/#!/primary//jsonschema/mediawiki/client/error mediawiki/client/error] events from external clients at https://intake-logging.wikimedia.org/v1/events.  eventgate-logging produces events to the Kafka logging cluster.


==== Clients ====
* Visibility: internal, submissions restricted.
* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/master/modules/ext.wikimediaEvents/clientError.js WikimediaEvents]
* Schemas: bundled, from local filesystem.
* [https://github.com/wikimedia/wikipedia-kaios/blob/master/src/utils/sendErrorLog.js Wikipedia KaiOS app]


== Development in Mediawiki Vagrant ==
The eventgate-main cluster produces events to the Kafka "main" clusters in both Eqiad and Codfw. It is used for low(ish) volume but high-priority eventsThese events are necessary for functioning of Wikimedia core services, like the MediaWiki [[Kafka Job Queue|Job Queue]] and change-propagation.
Installation of an EventGate service in Mediawiki vagrant is included via <tt>role::eventbus</tt>To install it, first edit <tt>puppet/hieradata/common.yaml</tt> and add:


<source lang=yaml>
Events are submitted here by:
npm::node_version: 10
*[[gerrit:plugins/gitiles/mediawiki/extensions/EventBus/+/master|EventBus MediaWiki extension]].  ([[gerrit:plugins/gitiles/operations/mediawiki-config/+/master/wmf-config/ProductionServices.php#45|Config]])
</source>
*[[gerrit:plugins/gitiles/operations/puppet/+/production/hieradata/role/common/scb.yaml#148|change-propagation]]
*[[gerrit:plugins/gitiles/operations/puppet/+/production/hieradata/role/common/restbase/production.yaml#21|restbase]] (resource_change events for purges)


===eventgate-analytics===
* Visibility: internal, submissions restricted.
* Schemas: bundled, from local filesystem AND dynamic, via via https://schema.wikimedia.org
The eventgate-analytics cluster produces events to the Kafka "jumbo" cluster, and is intended for high volume but low-priority events.  Events produced to eventgate-analytics should not be required for functional production services.  The Kafka jumbo cluster only exists in Eqiad, and does not have cross-dc replication and no cross-dc failover mode.  Events originating from Codfw are produced directly to Kafka "jumbo-eqiad".
Events are submitted here by:
*[[gerrit:plugins/gitiles/mediawiki/extensions/EventBus/+/master|EventBus MediaWiki extension]] (server-side PHP)  ([[gerrit:plugins/gitiles/operations/mediawiki-config/+/master/wmf-config/ProductionServices.php#44|Config]])
*WDQS Query Logging.  ([[gerrit:plugins/gitiles/operations/puppet/+/production/hieradata/role/common/wdqs.yaml#12|Config]])
*analytics/refinery Oozie util/swift/upload workflow.  ([[gerrit:plugins/gitiles/analytics/refinery/+/master/oozie/util/swift/upload/workflow.xml#57|Config]]) and Discovery/Search's glent oozie jobs (as of 2020-02).
*Discovery/Search's airflow [[gerrit:plugins/gitiles/wikimedia/discovery/analytics/+/master/airflow/plugins/swift_upload_plugin.py|swift upload plugin]].
===eventgate-analytics-external===
* Visibility: public, at https://intake-analytics.wikimedia.org/v1/events
* Schemas: dynamic, via via https://schema.wikimedia.org
The eventgate-analytics-external cluster produces events to the Kafka jumbo cluster. This replaces [[Analytics/Systems/EventLogging|EventLogging Analytics]], and can receive and validate events from external clients (like the EventLogging service before it).
Events are submitted here by:
*[[gerrit:plugins/gitiles/mediawiki/extensions/EventLogging/+/master|EventLogging MediaWiki extension]]
*[[gerrit:plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/master|WikimediaEvents extension]] (via EventLogging)
*[[gerrit:c/mediawiki/extensions/Popups/+/422261|Popups extension]] (TODO: Use EventLogging)
* Other event intake service clients, e.g. mobile apps, etc.
===eventgate-logging-external===
* Visibility: public, at https://intake-logging.wikimedia.org/v1/events
* Schemas: bundled, from local filesystem.
The eventgate-logging produces events to the Kafka "logging" cluster. eventgate-logging-external accepts [https://schema.wikimedia.org/#!/primary//jsonschema/mediawiki/client/error mediawiki/client/error] events from external clients. 
Events are submitted here by:
*[[gerrit:plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/master/modules/ext.wikimediaEvents/clientError.js|WikimediaEvents extension]] (via EventLogging)
*[https://github.com/wikimedia/wikipedia-kaios/blob/master/src/utils/sendErrorLog.js Wikipedia KaiOS app]
== Event Stream Config ==
EventGate in production will request stream configuration from the EventStreamConfig MediaWiki API.  Each service cluster restricts the stream configuration it uses via the <tt>destination_event_service</tt> setting; only streams that have <tt>destination_event_service</tt> matching the EventGate service cluster name (e.g. eventgate-main) will be used by that EventGate service cluster.
See also: [[Event_Platform/Stream_Configuration]]
==Validation Errors==
All EventGate clusters at WMF are configured to send validation error events to Logstash.  These errors are routed via Kafka and then ingested into Hive into the various <code>event.eventgate_*_error_validation</code> tables but also ingested into Logstash and viewable in Kibana.
[https://logstash.wikimedia.org/app/kibana#/dashboard/AXN5OoJu3_NNwgAUlbUT Logstash/Kibana dashboard: eventgate-validation]
[https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos Grafana EventGate dashboard]
== Local development ==
=== eventgate-wikimedia-dev.js ===
The eventgate-wikimedia codebase comes with an EventGate development implementation in eventgate-wikimedia-dev.js.  To use it, <tt>npm install --no-optional</tt> and then run <tt>./eventgate-wikimedia-dev.js</tt>.  Its default EventGate config is in <tt>./config.dev.yaml</tt>.
=== MediaWiki Vagrant ===
Installation of an EventGate service in MediaWiki-Vagrant is included via <code>role::eventbus</code>. 
EventGate requires NodeJS >= 10, but other NodeJS Mediawiki services are stuck on NodeJS 6 (for now). Ensuring that Node 10 is installed is required but may cause other NodeJS services to break.
EventGate requires NodeJS >= 10, but other NodeJS Mediawiki services are stuck on NodeJS 6 (for now). Ensuring that Node 10 is installed is required but may cause other NodeJS services to break.


Once done, you can enable the role and provision vagrant:
Install NodeJS 10, and enable the role, and provision vagrant:


<source lang=bash>
<syntaxhighlight lang="bash">
$ vagrant roles enable eventbus
$ vagrant hiera npm::node_version 10
$ vagrant roles enable eventbus  
$ vagrant provision
$ vagrant provision
</source>
</syntaxhighlight>


This will configure Mediawiki with the EventBus extension producing events to EventGate. The eventgate-wikimedia code repository will be cloned and the service will be launched from it using service-runner and a puppet installed config.vagrant.yaml. If you want to run the service manually (instead of letting systemd do it), you may stop it and run it with:
This will configure MediaWiki with the EventBus extension producing events to EventGate. The eventgate-wikimedia code repository will be cloned and the service will be launched from it using service-runner and a puppet installed config.vagrant.yaml. If you want to run the service manually (instead of letting systemd do it), you may stop it and run it with:
   nodejs /vagrant/srv/eventgate-wikimedia/node_modules/eventgate/server.js -c /vagrant/srv/eventgate-wikimedia/config.vagrant.yaml
   nodejs /vagrant/srv/eventgate-wikimedia/node_modules/eventgate/server.js -c /vagrant/srv/eventgate-wikimedia/config.vagrant.yaml


You might want to do this if you are developing EventGate code and want to run the process in the foreground.
You might want to do this if you are developing EventGate code and want to run the process in the foreground.


The eventgate-wikimedia repository contains the WMF specifc implementation of EventGate. It specifies the eventgate package as an npm depdenencyIf you want to modify the behavior of the eventgate library/service, then you will need to either clone https://github.com/wikimedia/eventgate and edit, or edit node_modules/eventgate.
Valid events will be written to <tt>/vagrant/logs/eventgate-events.json</tt> and eventgate-wikimedia errors (including validation errors) will be written to <tt>/vagrant/logs/eventgate-wikimedia.log</tt>You can pretty print the output of the eventgate-wikimedia log like
 
<syntaxhighlight lang="bash">
tail -f /vagrant/logs/eventgate-wikimedia.log | /vagrant/srv/eventgate-wikimedia/node_modules/bunyan/bin/bunyan
</syntaxhighlight>
 
==External links==
 
*[https://github.com/wikimedia/eventgate#readme Source code of EventGate (GitHub)]
*[[git:eventgate-wikimedia/|Source code of eventgate-wikimedia]]
*[[git:operations/deployment-charts/+/HEAD/charts/eventgate|deployment-charts/eventgate]]
*Schemas used by EventGate: from source: [[git:schemas/event/primary|primary]] and [[git:schemas/event/secondary|secondary]] or on wiki [[Event Platform/Schemas]].
 
[[Category:Services]]
[[Category:Event Platform]]

Revision as of 08:59, 8 October 2021


EventGate is an HTTP service for ingestion of events, written in Node.js. It takes JSON messages over HTTP POST requests, optionally validates them against a JSONSchema, and then produces them to a backend. The default backend (and the one used at WMF) is Kafka.

For more information about the EventGate codebase, see the README at github.com/wikimedia/eventgate. EventGate is meant to be generic and not WMF specific. It can be used standalone as a library, or it can be used with the built-in support for running Express (the Node.js HTTP server provided via mediawiki-service-template). WMF operates EventGate using the Express Node.js HTTP server.

This page documents the WMF deployments of EventGate.

Purpose

Event Platform seeks to standardize the process of producing and consuming event streams at WMF. EventGate is a an HTTP proxy for producing events that follows our practices. Specifically, it does the following:

  • Receives events of a specific schema version destined to a specific stream.
  • Uses EventStreamConfig to ensure that only events of the same schema lineage are emitted to a stream.
  • Augments the event with some custom defaults (e.g. setting date time fields or HTTP header data in the event).
  • Looks up the schema of the incoming event, and validates that the event data conforms to the schema.
  • Adds datacenter prefixes to the destination stream name to make the destination Kafka topic name.
  • Produces to Kafka.

Service

Main article: Event Platform/EventGate/Administration


Implementation

EventGate code is primarily hosted on Github for greater exposure to non-Wikimedia developers. There exists plenty of tooling around using Kafka with Avro but not much for Kafka with JSONSchemas. By hosting on Github we hope to gain more visibility and participation from non-WMF developers.

The eventgate-wikimedia repository contains the WMF deployment of EventGate. It uses the eventgate package as an npm dependency, and has additional utilities, configuration and deployment pipeline code for WMF's instances of EventGate.

To modify the behavior of the eventgate library or service, use EventGate directly instead, from https://github.com/wikimedia/eventgate.

Operation

There are multiple clusters of EventGate services in Wikimedia production.

EventGate is hosted in production as a containerized service running on Kubernetes, as such there are no Puppet roles or persistent backend hostnames.

As of Feb 2020, there are currently the following clusters (per deployment-charts):

  • eventgate-main
  • eventgate-analytics
  • eventgate-analytics-external
  • eventgate-logging-external

More details on how are used further down below.

Wikimedia EventGate configuration

Wikimedia's EventGate wrapper implements custom validate and produce behaviours for our use in WMF production. This includes configuration to look up JSON Schemas from either a the local filesystem (for eventgate-main) or a remote schema registry URL (for eventgate-analytics).

EventGate expects that specific implementations know how to map from an individual event to its JSONSchema. We use the $schema field in each of our JSON events to do this. This field contains a relative and versioned URI to the event's JSONSchema. EventGate fetches and caches this schema and uses it to validate each event with the same $schema.

Producer types: Guaranteed and Hasty

Wikimedia's EventGate configuration offers two different Kafka producer connections, named guaranteed and hasty.

The guaranteed producer will block the HTTP response until the event has been validated and sent to the Kafka brokers with either an ACK response or known failure to ACK. Note that "guaranteed" does not mean that the event is guaranteed to be persisted in Kafka (there is not an indefinite retry or other buffer). Rather it means that HTTP response status can be trusted, so a 2xx status code guarantees that the event has been persisted.

The hasty producer is optimised for high-throughput, and will not block the HTTP response. Instead, it will immediately return a 202 status response as soon as EventGate has received the JSON message from the HTTP response body. The event will be validated and produced to Kafka afterward. The HTTP client that submitted the event will not know whether the event was valid or whether it was succesfully persisted in Kafka. If the event failed validation or failed production to Kafka, an error will be logged to Logstash however.

The guaranteed producer type is the default for the /v1/events endpoint. To POST an event in hasty mode, set hasty=true in the request query parameters.

EventGate clusters

At WMF, EventGate is deployed as multiple separate clusters, each with its own defined purpose.

All EventGate clusters are open to receive events from any internal production server. MediaWiki produces to EventGate using the EventBus extension. (Apologies if this is confusing! See Event* for a disambiguation page for related terms.)

eventgate-main

  • Visibility: internal, submissions restricted.
  • Schemas: bundled, from local filesystem.

The eventgate-main cluster produces events to the Kafka "main" clusters in both Eqiad and Codfw. It is used for low(ish) volume but high-priority events. These events are necessary for functioning of Wikimedia core services, like the MediaWiki Job Queue and change-propagation.

Events are submitted here by:

eventgate-analytics

The eventgate-analytics cluster produces events to the Kafka "jumbo" cluster, and is intended for high volume but low-priority events. Events produced to eventgate-analytics should not be required for functional production services. The Kafka jumbo cluster only exists in Eqiad, and does not have cross-dc replication and no cross-dc failover mode. Events originating from Codfw are produced directly to Kafka "jumbo-eqiad".

Events are submitted here by:

eventgate-analytics-external

The eventgate-analytics-external cluster produces events to the Kafka jumbo cluster. This replaces EventLogging Analytics, and can receive and validate events from external clients (like the EventLogging service before it).

Events are submitted here by:

eventgate-logging-external

The eventgate-logging produces events to the Kafka "logging" cluster. eventgate-logging-external accepts mediawiki/client/error events from external clients.

Events are submitted here by:

Event Stream Config

EventGate in production will request stream configuration from the EventStreamConfig MediaWiki API. Each service cluster restricts the stream configuration it uses via the destination_event_service setting; only streams that have destination_event_service matching the EventGate service cluster name (e.g. eventgate-main) will be used by that EventGate service cluster.

See also: Event_Platform/Stream_Configuration

Validation Errors

All EventGate clusters at WMF are configured to send validation error events to Logstash. These errors are routed via Kafka and then ingested into Hive into the various event.eventgate_*_error_validation tables but also ingested into Logstash and viewable in Kibana.

Logstash/Kibana dashboard: eventgate-validation

Grafana EventGate dashboard

Local development

eventgate-wikimedia-dev.js

The eventgate-wikimedia codebase comes with an EventGate development implementation in eventgate-wikimedia-dev.js. To use it, npm install --no-optional and then run ./eventgate-wikimedia-dev.js. Its default EventGate config is in ./config.dev.yaml.

MediaWiki Vagrant

Installation of an EventGate service in MediaWiki-Vagrant is included via role::eventbus. EventGate requires NodeJS >= 10, but other NodeJS Mediawiki services are stuck on NodeJS 6 (for now). Ensuring that Node 10 is installed is required but may cause other NodeJS services to break.

Install NodeJS 10, and enable the role, and provision vagrant:

$ vagrant hiera npm::node_version 10
$ vagrant roles enable eventbus 
$ vagrant provision

This will configure MediaWiki with the EventBus extension producing events to EventGate. The eventgate-wikimedia code repository will be cloned and the service will be launched from it using service-runner and a puppet installed config.vagrant.yaml. If you want to run the service manually (instead of letting systemd do it), you may stop it and run it with:

 nodejs /vagrant/srv/eventgate-wikimedia/node_modules/eventgate/server.js -c /vagrant/srv/eventgate-wikimedia/config.vagrant.yaml

You might want to do this if you are developing EventGate code and want to run the process in the foreground.

Valid events will be written to /vagrant/logs/eventgate-events.json and eventgate-wikimedia errors (including validation errors) will be written to /vagrant/logs/eventgate-wikimedia.log. You can pretty print the output of the eventgate-wikimedia log like

tail -f /vagrant/logs/eventgate-wikimedia.log | /vagrant/srv/eventgate-wikimedia/node_modules/bunyan/bin/bunyan

External links