Event Platform/Instrumentation How To

Event Platform

Documentation

• Schemas
• Schema guidelines
• Event Stream Configuration
• Producer requirements

---

• Stream Processing/Flink
• Stream Processing Use Cases
• Stream Processing Framework Evaluation

---

• Instrumentation tutorial
• Event data sanitization in Hadoop
• Differences with EventLogging legacy

---

• Disambiguation page
• Maintainers

---

Services, libraries, and repositories

• Event Utilities (code client libraries)
• EventGate service (admin)
• EventStreams service (admin)
• Schema API
• EventStreams API
• EventStreams API (Beta Cluster)
• Primary event schema repo
• Secondary event schema repo

See also

• EventBus extension (mediawiki.org)
• EventLogging extension (mediawiki.org)
• Issue tracker (Phabricator)
• Event Platform epic task (Phabricator)

[]

Wikimedia's Event Platform supports both production "Tier 1" events, as well as analytics "Tier 2" events. To support both, the backend systems that handle these events are more reliable and scalable, but also more complex than legacy EventLogging. This page will provide an instrumentation event stream lifecycle example for developing, deploying, evolving, and decommissioning MediaWiki client side JS instrumentation event streams from the mw:Extension:WikimediaEvents via mw:Extension:EventLogging.

You can learn more about the differences between how Event Platform works and how the legacy EventLogging backend systems work on the EventLogging legacy page.

Event Streams and Schemas

Both event schemas and streams must be declared before they can be used by EventLogging. Doing so will require three changes:

• To create or modify a schema, you will create and edit a current.yaml JSONSchema file in the schemas/event/secondary repository. You can read more in depth about how Event Platform schemas work at Event_Platform/Schemas. Please also read Event_Platform/Schemas/Guidelines before creating or modifying scheams.

• To declare an event stream, you will edit the mediawiki-config repository and add a stream config entry to $wgEventStreams (in wmf-config/InitialiseSettings.php and/or wmf-config/InitialiseSettings-labs.php).

• To tell EventLogging that it should look up stream config for a stream (and that it is allowed to produce that stream), you will add an entry to $wgEventLoggingStreamNames

Instrumentation Event Stream Lifecycle Example

Event Platform's instrumention development lifecycle is still a work in progress, so please be patient as we work to improve this. Feedback and ideas are very welcome!

Assuming you'll be using the WikimediaEvents extension to produce your events, you'll need to make changes to 3 git repositories: schemas/events/secondary, WikimediaEvents, and finally for deployment (in beta and production) mediawiki-config.

This lifecycle example will demonstrate creating a new stream to log whenever a user hovers over an interwiki link. We'll create a new event stream called 'mediawiki.interwiki_link_hover' that conforms to a new 'analytics/link_hover' schema.

Development

In MediaWiki Docker

See MediaWiki Docker EventLogging Configuration recipe.

In MediaWiki Vagrant

Enabling the wikimediaevents role will also include the eventlogging role for you, and set up other Event Platform backend components on MediaWiki Vagrant including EventGate.

$ vagrant roles enable wikimediaevents --provision
$ vagrant git-update

This will clone WikimediaEvents into mediawiki/extensions/WikimediaEvents and the schemas/event/secondary repository at srv/schemas/event/secondary (and also install its npm dependencies for schema materialization and tests)

MediaWiki Vagrant's EventLogging and EventGate setup will allow events of any schema into any stream. For now, you will not have to think about stream config in your development environment.

Events will be written to /vagrant/logs/eventgate-events.json. eventgate logs, including validation errors, are in /vagrant/logs/eventgate-wikimedia.log.

To verify that eventgate is working properly, you can force a test event to be produced by curl-ing http://localhost:8192/v1/_test/events. You should see a test event logged into eventgate-events.json.

In your local dev environment with eventgate-devserver

If you aren't using Mediawiki-Vagrant, or you'd rather have more manual control over your development environment, EventLogging comes with an 'eventgate-devserver' that will accept events and write them to a local file. Clone mediawiki/extensions/EventLogging and run

$ cd extensions/EventLogging/devserver
$ npm install --skip-optional
$ npm run eventgate-devserver

This should download EventGate and other dependencies and run the eventgate-devserver accepting events at http://localhost:8192 and writing them to ./events.json. See the devserver/README.md for more info.

Creating a new schema

EventLogging instrumentation schemas will be added to the schemas/event/secondary repository in the jsonschema/analytics namespace. We want to create a new schema that represents a link hover event. Because this type of event can be modeled pretty generically, we are going to create a generic link hover event schema, not one that is specific to interwiki links. This will allow the schema to possibly be reused by other types of link hover events.

Create a new file in schemas/event/secondary at jsonschema/analytics/link_hover/current.yaml:

$ cd srv/schemas/event/secondary
$ mkdir -p jsonschema/analytics/link_hover

Create jsonschema/analytics/link_hover/current.yaml with this content:

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.0.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link

examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"mediawiki.interwiki_link_hover"},"dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging"}

NOTE: Event_Platform/Schemas/Guidelines has rules and conventions for schemas.

Now git add and git commit this new file. jsonschema-tools will automatically dereference and materialize this file for you.

$ git add jsonschema/analytics/link_hover/current.yaml

$ git commit -m 'Add new analytics/link_hover schema'

[2020-05-20 18:47:33.865 +0000]: Looking for modified current.yaml schema files in ./jsonschema/
[2020-05-20 18:47:33.946 +0000]: Materializing ./jsonschema/analytics/link_hover/current.yaml...
[2020-05-20 18:47:33.957 +0000]: Dereferencing schema with $id /analytics/link_hover/1.0.0 using schema base URIs ./jsonschema/,https://schema.wikimedia.org/repositories/primary/jsonschema/
[2020-05-20 18:47:34.005 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.0.0.yaml.
[2020-05-20 18:47:34.006 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.0.0.json.
[2020-05-20 18:47:34.007 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.yaml -> 1.0.0.yaml.
[2020-05-20 18:47:34.008 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.json -> 1.0.0.json.
[2020-05-20 18:47:34.008 +0000]: Created extensionless symlink ./jsonschema/analytics/link_hover/1.0.0 -> 1.0.0.yaml.
[2020-05-20 18:47:34.009 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest -> 1.0.0.yaml.
[2020-05-20 18:47:34.011 +0000]: New schema files have been materialized. Adding them to git: ./jsonschema/analytics/link_hover/1.0.0.yaml,./jsonschema/analytics/link_hover/latest.yaml,./jsonschema/analytics/link_hover/1.0.0,./jsonschema/analytics/link_hover/latest,./jsonschema/analytics/link_hover/1.0.0.json,./jsonschema/analytics/link_hover/latest.json
[analytics_link_hover bd4496b] Add new analytics/link_hover schema
 7 files changed, 248 insertions(+)
 create mode 120000 jsonschema/analytics/link_hover/1.0.0
 create mode 100644 jsonschema/analytics/link_hover/1.0.0.json
 create mode 100644 jsonschema/analytics/link_hover/1.0.0.yaml
 create mode 100644 jsonschema/analytics/link_hover/current.yaml
 create mode 120000 jsonschema/analytics/link_hover/latest
 create mode 120000 jsonschema/analytics/link_hover/latest.json
 create mode 120000 jsonschema/analytics/link_hover/latest.yaml

As you can see, you now have many more files committed to git than just current.yaml. Event Platform will use the statically versioned schema files to validate your events.

If you want to materialize a current.yaml schema file without committing it to git, you run the command to materialize it manually:

$ ./node_modules/.bin/jsonschema-tools materialize jsonschema/analytics/link_hover/current.yaml

You can read more about how and why we materialize schema versions over at Event_Platform/Schemas.

Writing MediaWiki instrumentation code using the EventLogging extension

In JavaScript

Assuming you already have a ResourceLoader package module in WikimediaEvents, you need to add code that calls the mw.eventLog.submit function. We want to fire events whenever an interwiki link is hovered over.

// 'a.extiw' will match anchors that have a extiw class.  extiw is used for interwiki links.
$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		$schema: '/analytics/link_hover/1.0.0',
		link_href: link.href,
		link_title: link.title,
	};
	var streamName = 'mediawiki.interwiki_link_hover';
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Now when you over over a link, mw.eventLog.submit will be called and an event will be sent to the 'mediawiki.interwiki_link_hover' stream.

In summary:

• Make sure your event data includes $schema which should match $id and is set to the path (starting with /) and (extensionless) version. This tells EventGate which schema and specifically which version of that schema the instrumentation conforms to.
• mw.eventLog.submit needs: (1) the stream name (this must match what will be configured in production in wgEventStreams stream config, and (2) the event data must include $schema.

In PHP

The EventLogging PHP interface is the same as the JavaScript one. If you were building a server side event that will be send with the EventLogging extension, you'd use the EventLogging::submit function.

$exampleEvent = [
	'$schema' => '/analytics/example_schema/1.0.0'
	'field_a' => 'value_a'
	// ... Other event data fields from /analytilcs/example_schema/1.0.0
];

$streamName = 'mediawiki.example.stream';
EventLogging::submit( $streamName, $exampleEvent );

Deployment

Once your schema and instrumentation code have been reviewed and merged, you are ready for deployment. You will make some changes to the mediawiki-config repository to configure your new stream, as well as register it for use by the EventLogging extension.

Clone mediawiki-config and edit wmf-config/InitialiseSettings.php. NOTE: You can configure these same settings for beta only by editing wmf-config/InitialiseSettings-labs.php instead. InitialiseSettings.php will be used in both beta and production if no corresponding values are found in InitialiseSettings-labs.php.

Stream Configuration

First declare your stream in the wgEventStreams config variable.

'wgEventStreams' => [
	'default' => [
		// ...
		'mediawiki.interwiki_link_hover' => [
			'schema_title' => 'analytics/link_hover',
			'destination_event_service' => 'eventgate-analytics-external',
		],
	],
],

This is the minimal stream config required. This config is used to ensure that only events that have a schema with a title that matches the schema_title value here are allowed in the stream. (Note that this is NOT the same as the schema's $id field. The $id field is a versioned URI. Each event data's $schema URI field will be used to lookup the schema at that schema URI.)

Future work will add support for other stream configs, like adjusting sampling rate without having to deploy code.

See Event_Platform/Stream_Configuration#Common_Settings_Documentation for documentation on common stream config settings.

Register your stream for use by EventLogging

If producing events using the EventLogging extension (most likely) You need to list your stream in wgEventLoggingStreamNames so that EventLogging will get the config for your stream and be able to produce these events.

'wgEventLoggingStreamNames' => [
	'default' => [
		// ...
		'mediawiki.interwiki_link_hover',
	],
],

If you've made these changes in InitialiseSettings-labs.php, you can find a reviewer to just merge your change and the config will be automatically synced to the beta cluster. If your schema change and instrumentation code change are also merged, you'll be able to send these events in beta.

If you've made these changes in InitialiseSettings.php, you'll need to schedule a Backport window deployment to sync out your config change to the production cluster. See Deployments and Backport windows for instructions.

Viewing and querying events

In beta

Event Platform components are set up in the Cloud VPS deployment-prep project (AKA 'beta') similar to how they run in production. The EventLogging extension there is configured to send events to https://intake-analytics.wikimedia.beta.wmflabs.org/v1/events?hasty=true. If you are not using EventLogging (e.g. from a mobile app), you will have to configure your 'beta' installation of the app to also POST events to this URL.

An EventStreams instance in the Cloud VPS deployment-prep project (AKA 'beta') exists that allows you to consume any stream declared in stream config. As this instance is in beta, it only has events that are generated in beta. You can consume this stream via an EventSource/SSE client (see https://stream-beta.wmflabs.org/?doc) or in your browser using the EventStreams GUI at https://stream-beta.wmflabs.org/v2/ui.

Alternatively, you can log into a host in the deployment-prep Cloud VPS project and consume events directly from Kafka using a Kafka client like kafkacat:

sudo apt-get install kafkacat # if needed
kafkacat -C -b deployment-kafka-jumbo-2.deployment-prep.eqiad1.wikimedia.cloud -t eqiad.<your_stream_name>

In production

Events in production eventually make their way into the Analytics/Data_Lake where they are ingested into a Hive table in the event Hive database. From there they are queryable using Hive, Spark, or Presto, and also available for dashboarding in Superset via Presto.

The Hive table name will be a normalized version of the stream name. Our example stream's Hive table will be event.mediawiki_interwiki_link_hover.

Events are also available realtime by consuming them directly from Kafka. The stream name is prefixed with the source datacenter for the Kafka topic. WMF has two main datacenters: 'eqiad' and 'codfw'. To consume all events from Kafka for your stream, you should consume from both of these topics. Our example's Kafka topics will be 'eqiad.mediawiki.interwiki_link_hover and 'codfw.mediawiki.interwiki_link_hover'.

An internal only EventStreams instance exists in production that allows you to consume any stream declared in stream config. eventstreams-internal.discovery.wmnet is not publicly accessible; you can only access it through an ssh tunnel, e.g.:

# Tunnel local traffic to port 4992 to eventstreams-internal.discovery.wmnet:4992 via bast1003.wikimedia.org
# Replace bast1003.wikimedia.org with your preferred bastion or other accessible production host.
ssh -N -L4992:eventstreams-internal.discovery.wmnet:4992 bast1003.wikimedia.org

Then, in your browser, you can navigate to https://localhost:4992/v2/ui/#/ to use the GUI, or you can consume with your own EventSource/SSE client (see https://localhost:4992/?doc).

Viewing schema validation errors

Events that do not validate with their schema will not be produced. For events that are produced via EventLogging, validation error events will be produced into the stream eventgate-analytics-external.error.validation and will be ingested into Hive in the event.eventgate_analytics_external_error_validation table.

These validation error events are also ingested into logstash and can be viewed in Kibana using this dashboard.

The rate of validation errors per stream can be seen in the EventGate Grafana dashboard.

The eventgate-analytics-external.error.validation stream is a stream like any other, so you can also view the stream using an EventStreams GUI in beta or production (see above).

Evolving

Let's say our 'mediawiki.interwiki_link_hover' stream is operating fine in production, and now we want to also log the link's text value. We'll need to add a new link_text field to the 'analytics/link_hover' schema.

Edit jsonschema/analytics/link_hover/current.yaml again and add the field and bump the version number in the $id. The new content of current.yaml should look like:

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.1.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link
      link_text:
        type: string
        description: text value of http anchor link
examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"mediawiki.interwiki_link_hover"},"dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging", "link_text": "EventLogging extension"}

Now, add and commit current.yaml again and jsonschema-tools will materialize new 1.1.0 version files.

$ git add jsonschema/analytics/link_hover/current.yaml
$ git commit -m 'analytics/link_hover - add link_text field and bump to version 1.1.0'
[2020-05-20 19:52:39.162 +0000]: Looking for modified current.yaml schema files in ./jsonschema/
[2020-05-20 19:52:39.229 +0000]: Materializing ./jsonschema/analytics/link_hover/current.yaml...
[2020-05-20 19:52:39.235 +0000]: Dereferencing schema with $id /analytics/link_hover/1.1.0 using schema base URIs ./jsonschema/,https://schema.wikimedia.org/repositories/primary/jsonschema/
[2020-05-20 19:52:39.264 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.1.0.yaml.
[2020-05-20 19:52:39.271 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.1.0.json.
[2020-05-20 19:52:39.275 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.yaml -> 1.1.0.yaml.
[2020-05-20 19:52:39.275 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.json -> 1.1.0.json.
[2020-05-20 19:52:39.276 +0000]: Created extensionless symlink ./jsonschema/analytics/link_hover/1.1.0 -> 1.1.0.yaml.
[2020-05-20 19:52:39.278 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest -> 1.1.0.yaml.
[2020-05-20 19:52:39.279 +0000]: New schema files have been materialized. Adding them to git: ./jsonschema/analytics/link_hover/1.1.0.yaml,./jsonschema/analytics/link_hover/latest.yaml,./jsonschema/analytics/link_hover/1.1.0,./jsonschema/analytics/link_hover/latest,./jsonschema/analytics/link_hover/1.1.0.json,./jsonschema/analytics/link_hover/latest.json
[analytics_link_hover 3c4f8c7] analytics/link_hover - add link_text field and bump to version 1.1.0
 7 files changed, 227 insertions(+), 4 deletions(-)
 create mode 120000 jsonschema/analytics/link_hover/1.1.0
 create mode 100644 jsonschema/analytics/link_hover/1.1.0.json
 create mode 100644 jsonschema/analytics/link_hover/1.1.0.yaml

New analytics/link_hover/1.1.0 files have been created, and the jsonschema/analytics/link_hover/latest symlinks have been updated to point to the 1.1.0 versions.

Now edit your instrumentation code to produce the event data with the link_text field and the updated versioned $schema URI. Your instrumentation code should now look like this:

$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		// We've added link_text, which was added in schema verison 1.1.0, so we need to specify that this
		// event should be validated using schema verison 1.1.0.
		$schema: '/analytics/link_hover/1.1.0',
		link_href: link.href,
		link_title: link.title,
		link_text: link.text,
	}
	var streamName = 'mediawiki.interwiki_link_hover'
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Note that only backwards compatible changes are allowed. This means that the only type of change you can do to a schema is add new optional fields. jsonschema-tools will ensure that all schema versions are backwards compatible (as well as ensuring that the schema repository is in good shape). Jenkins will run CI tests when you push your schema change to gerrit, but you can also run the tests manually:

$ npm test
...
  Schema Compatibility in Repository ./jsonschema/
    analytics/link_hover
      Major Version 1
        ✓ 1.1.0 must be compatible with 1.0.0
...

Overriding event stream config settings

In production for specific wikis / groups

Because EventStreamConfig uses mediawiki-config, we can override settings for beta, wiki groups, or even specific wikis. Unfortunetely, this ONLY works for clients that request their config via the specific wiki you are overriding.

EventGate is a service that is not wiki specific, so it requests all stream configs from meta.wikimedia.org. This means that you MUST add entries in wgEventStreams for at least metawiki (usually done so in the 'default' configs).

EventLogging is a MediaWiki extension that produces events, and is usually executed within the context of a specific wiki. It will get configuration for the specific wiki it is being run in. That means you can override configs that EventLogging uses for specific wikis.

Example: EventLogging respects a sample setting only produce a sample of events. We only want to send a 1/10 events on enwiki. We've already got the mediawiki.interwiki_link_hover stream declared in wgEventStreams default. Let's add an override for the sample setting on enwiki.

'wgEventStreams' => [
	'default' => [
		// ...
		'mediawiki.interwiki_link_hover' => [
			'schema_title' => 'analytics/link_hover',
			'destination_event_service' => 'eventgate-analytics-external',
		],
	],

    // Add the overrides for enwiki.
    // The '+' prefix indicates that these settings should be recursively merged with 'default' settings.
    // See: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/src/StaticSiteConfiguration.php#64
    '+enwiki' => [
        'mediawiki.interwiki_link_hover' => [
            'sample' => [
                'rate' => 0.1,
            ],
        ],
    ]
],

When enwiki MediaWiki JavaScript requests event stream config for the mediawiki.interwiki_link_hover stream, it will get something like

{
  "streams": {
    "mediawiki.interwiki_link_hover": {
      "schema_title": "analytics/link_hover",
      "destination_event_service": "eventgate-analytics-external",
      "sample": {
        "rate": 0.1,
      },
    }
  }
}

with the default config merged together.

In beta

Beta AKA deployment-prep configs are in InitialiseSettings-labs.php. For the most part, overrides here work exactly as described above. However, the 'default' configs will not be merged if you set them. That means that you cannot use the 'default' config section in wgEventStreams in InitialiseSettings-labs.php. You can however merge overrides in from InitialiseSettings.php 'default' section if you use the per wiki or group merge override as described above. E.g. adding the following in InitialiseSettings-labs.php will override the sample rate for en.wikipedia.beta.wmflabs.org:

'wgEventStreams' => [
     
    // Add the overrides for enwiki in beta
    // The '+' prefix indicates that these settings should be recursively merged with 'default' settings from InitialiseSettings.php.
    '+enwiki' => [
        'mediawiki.interwiki_link_hover' => [
            'sample' => [
                'rate' => 1.0,
            ],
        ],
    ]
],

Decommissioning

When deploying a new instrumentation event stream, you should also plan to decommission it one day. A schema should never be deleted, but all of the stream related code and configuration can be removed at anytime to stop producing an event stream. In the rare cases where the presence of the schema itself is problematic, we can delete it but it requires a coordinated effort to exclude it and prevent alarms along the data pipeline.