Event Platform/Instrumentation How To

Wikimedia's Event Platform supports both production 'tier 1' events, as well as analytics 'tier 2' events. To support both, the backend systems that handle these events are more reliable and scalable, but also more complex than legacy EventLogging. This page will provide an instrumentation event stream lifecycle example for developing, deploying, evolving, and decommissioning MediaWiki client side JS instrumentation event streams from the mw:Extension:WikimediaEvents via mw:Extension:EventLogging.

Differences from EventLogging the legacy backend

The EventLogging extension was originally built as an all in one system to capture MediaWiki analytics events. It managed schemas, client side event submission, server side event validation and server side event ingestion (into e.g. MySQL). The Event Platform program was conceived to unify event collection for production and analytics events. EventLogging's tier 2 and analytics focus and breadth was not suitable to support this unification. Many of the features of WMF's Event Platform are the same as the legacy EventLogging system, but are more modular and scalable. From an instrumentation only perspective, it may not be clear why things have to be different, but there are good engineering reasons for all of these changes.

The EventLogging extension has been repurposed as an MediaWiki instrumentation event producer library only. On wiki schemas and backend validation are no longer supported by EventLogging.

	EventLogging legacy	Event Platform
Schema repositories	EventLogging schemas were stored as centralized wiki pages on metawiki, and all environments (development, beta, production, etc.) had to use this same schema repository.	Event Platform schema are in decentralized git repositories. (Analytics instrumentation schemas are in the schemas/event/secondary repository. Schema repositories are also readable at https://schema.wikimedia.org/#!/ )
Streams, not schemas	EventLogging schemas were single use. Each schema corresponded to only one instrumentation, and eventually only one downstream SQL table.	Event Platform schemas are like data types for a dataset. A realtime event data set is called an 'event stream' (or just 'stream' for shorthand). Each stream must specify its schema, and a schema may be used by multiple streams.
Schema versions	EventLogging schema versions were wiki page revisions. Each event specified its schema name and revision.	Event Platform schemas are semantically versioned, and each event declares its schema and version in a $schema URI.
Schema compatibility	Each EventLogging schema revision could change the schema in any way, which lead to backwards incompatible changes.	Event Platform schemas versions must be backwards compatible; i.e. only adding new optional fields is allowed.
Stream config	None. Changes to the way events were emitted (like sampling rate) required a code deployment.	Streams are configured in mediawiki-config and can be modified via a Backport window deployment.

Event Streams and Schemas

Both event schemas and streams must be declared before they can be used by EventLogging. Doing so will require three changes:

• To create or modify a schema, you will create and edit a current.yaml JSONSchema file in the schemas/event/secondary repository. You can read more in depth about how Event Platform schemas work at Event_Platform/Schemas. Please also read Event_Platform/Schemas/Guidelines before creating or modifying scheams.

• To declare an event stream, you will edit the mediawiki-config repository and add a stream config entry to $wgEventStreams (in wmf-config/InitialiseSettings.php and/or wmf-config/InitialiseSettings-labs.php).

• To tell EventLogging that is should look up stream config for a stream (and that is is allowed to produce that stream), you will add an entry to $wgEventLoggingStreamNames

Instrumentation Event Stream Lifecycle Example

Event Platform's instrumention development lifecycle is still a work in progress, so please be patient as we work to improve this. Feedback and ideas are very welcome!

Assuming you'll be using the WikimediaEvents extension to produce your events, you'll need to make changes to 3 git repositories: schemas/events/secondary, WikimediaEvents, and finally for deployment (in beta and production) mediawiki-config.

This lifecycle example will demonstrate creating a new stream to log whenever a user hovers over an interwiki link. We'll create a new event stream called 'analytics.interwiki_link_hover' that conforms to a new 'analytics/link_hover' schema.

Development

Setup In MediaWiki Vagrant

Enabling the wikimediaevents role will also include the eventlogging role for you, and set up other Event Platform backend components on MediaWiki Vagrant including EventGate and Kafka.

$ vagrant roles enable wikimediaevents --provision
$ vagrant git-update

This will clone WikimediaEvents into mediawiki/extensions/WikimediaEvents and the schemas/event/secondary repository at srv/schemas/event/secondary (and also install its npm dependencies for schema materialization and tests)

Mediawiki Vagrant's EventLogging and EventGate setup will allow events of any schema into any stream. For now, you will not have to think about stream config in your development environment.

Creating a new schema

EventLogging instrumentation schemas will be added to the schemas/event/secondary repository in the jsonschema/analytics namespace. We want to create a new schema that represents a link hover event. Because this type of event can be modeled pretty generically, we are going to create a generic link hover event schema, not one that is specific to interwiki links. This will allow the schema to possibly be reused by other types of link hover events.

Create a new file in at in schemas/event/secondary at jsonschema/analytics/link_hover/current.yaml:

$ cd srv/schemas/event/secondary
$ mkdir -p jsonschema/analytics/link_hover

Create jsonschema/analytics/link_hover/current.yaml with this content:

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.0.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link

examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"analytics.interwiki_link_hover"},"event_dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging"}

NOTE: Event_Platform/Schemas/Guidelines has rules and conventions for schemas.

Now git add and git commit this new file. jsonschema-tools will automatically dereference and materialize this file for you.

$ git add jsonschema/analytics/link_hover/current.yaml

$ git commit -m 'Add new analytics/link_hover schema'

[2020-05-20 18:47:33.865 +0000]: Looking for modified current.yaml schema files in ./jsonschema/
[2020-05-20 18:47:33.946 +0000]: Materializing ./jsonschema/analytics/link_hover/current.yaml...
[2020-05-20 18:47:33.957 +0000]: Dereferencing schema with $id /analytics/link_hover/1.0.0 using schema base URIs ./jsonschema/,https://schema.wikimedia.org/repositories/primary/jsonschema/
[2020-05-20 18:47:34.005 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.0.0.yaml.
[2020-05-20 18:47:34.006 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.0.0.json.
[2020-05-20 18:47:34.007 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.yaml -> 1.0.0.yaml.
[2020-05-20 18:47:34.008 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.json -> 1.0.0.json.
[2020-05-20 18:47:34.008 +0000]: Created extensionless symlink ./jsonschema/analytics/link_hover/1.0.0 -> 1.0.0.yaml.
[2020-05-20 18:47:34.009 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest -> 1.0.0.yaml.
[2020-05-20 18:47:34.011 +0000]: New schema files have been materialized. Adding them to git: ./jsonschema/analytics/link_hover/1.0.0.yaml,./jsonschema/analytics/link_hover/latest.yaml,./jsonschema/analytics/link_hover/1.0.0,./jsonschema/analytics/link_hover/latest,./jsonschema/analytics/link_hover/1.0.0.json,./jsonschema/analytics/link_hover/latest.json
[analytics_link_hover bd4496b] Add new analytics/link_hover schema
 7 files changed, 248 insertions(+)
 create mode 120000 jsonschema/analytics/link_hover/1.0.0
 create mode 100644 jsonschema/analytics/link_hover/1.0.0.json
 create mode 100644 jsonschema/analytics/link_hover/1.0.0.yaml
 create mode 100644 jsonschema/analytics/link_hover/current.yaml
 create mode 120000 jsonschema/analytics/link_hover/latest
 create mode 120000 jsonschema/analytics/link_hover/latest.json
 create mode 120000 jsonschema/analytics/link_hover/latest.yaml

As you can see, you now have many more files committed to git than just current.yaml. Event Platform will use the statically versioned schema files to validate your events.

If you want to materialize a current.yaml schema file without committing it to git, you run the command to materialize it manually:

$ ./node_modules/.bin/jsonschema-tools materialize jsonschema/analytics/link_hover/current.yaml

You can read more about how and why we materialize schema versions over at Event_Platform/Schemas.

Event stream configuration

Event streams do not need to be configured to test instrumentation if using the debug mode, which can be enabled by logging in to an account and turning eventlogging-display-web user preference on:

mw.loader.using('mediawiki.api.options')
    .then(
        () => new mw.Api().saveOption('eventlogging-display-web', '1')
    );

You will see the events as notifications to check that they're generated when they should be and that they have the data they need to have. However, for event data to actually be sent, the stream needs to be configured because streams that are not in the stream configuration are never in-sample.

When developing with MediaWiki-Vagrant you do not need to specify sampling configuration for your streams, but you at least need to register them locally (and they will use 100% sampling by default). After enabling the "wikimediaevents" role and provisioning, add the following to LocalSettings.php in your clone:

$wgEventStreams = [
    [
        'stream' => 'analytics.link_hover',
        'schema_title' => 'analytics/link_hover',
        'destination_event_service' => 'eventgate-analytics-external',
    ],
];

$wgEventLoggingStreamNames = [
	'analytics.link_hover',
];

You should be able to check that the stream config works by querying your local MediaWiki API: http://dev.wiki.local.wmftest.net:8080/w/api.php?action=streamconfigs&format=json or http://dev.wiki.local.wmftest.net:8080/w/api.php?action=streamconfigs&format=json&constraints=destination_event_service=eventgate-analytics-external and you should see something like {"streams":{"analytics.link_hover":{"destination_event_service":"eventgate-analytics-external"}}}

Note: $wgEventStreams holds the stream configuration and $wgEventLoggingStreamNames registers streams with EventLogging, so that their configuration to EL. If your instrumentation does not appear to log events to a certain stream, check that the stream is in both of these.

Reminder: any streams you add/use locally will need to be added to production config by adding a stream config entry to $wgEventStreams (in wmf-config/InitialiseSettings.php and/or wmf-config/InitialiseSettings-labs.php in mediawiki-config) for your instrumentation to actually send events once deployed.

Writing instrumentation code

Assuming you already have a ResourceLoader package module in WikimediaEvents, you need to add code that calls the mw.eventLog.submit function. We want to fire events whenever an interwiki link is hovered over.

// 'a.extiw' will match anchors that have a extiw class.  extiw is used for interwiki links.
$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		$schema: '/analytics/link_hover/1.0.0',
		link_href: link.href,
		link_title: link.title,
	};
	var streamName = 'analytics.interwiki_link_hover';
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Now when you over over a link, mw.eventLog.submit will be called and an event will be sent to the 'analytics.interwiki_link_hover' stream.

In summary:

• Make sure your event data includes $schema which should match $id and is set to the path (starting with /) and (extensionless) version. This tells EventGate which schema and specifically which version of that schema the instrumentation conforms to.
• mw.eventLog.submit needs: (1) the stream name, as configured in wgEventStreams, and (2) the event data must include $schema.

Deployment

Once your schema and instrumentation code have been reviewed and merged, you are ready for deployment. You will make some changes to the mediawiki-config repository to configure your new stream, as well as register it for use by the EventLogging extension.

Clone mediawiki-config and edit wmf-config/InitialiseSettings.php. NOTE: You can configure these same settings for beta only by editing wmf-config/InitialiseSettings-labs.php instead. InitialiseSettings.php will be used in both beta and production if no corresponding values are found in InitialiseSettings-labs.php.

First add an entry to stream config in the wgEventStreams config variable.

'wgEventStreams' => [
	'default' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover'
		],
	],
],

This is the minimal stream config required. This config is used to ensure that only events that have a schema with a title that matches the schema_title value here are allowed in the stream. (Note that this is NOT the same as the schema's $id field. The $id field is a versioned URI. Each event data's $schema URI field will be used to lookup the schema at that schema URI.)

Future work will add support for other stream configs, like adjusting sampling rate without having to deploy code.

Now list your stream in wgEventLoggingStreamNames so that EventLogging will get the config for your stream and be able to produce these events.

'wgEventLoggingStreamNames' => [
	'default' => [
		// ...
		'analytics.interwiki_link_hover'
	],
],

If you've made these changes in InitialiseSettings-labs.php, you can find a reviewer to just merge your change and the config will be automatically synced to the beta cluster. If your schema change and instrumentation code change are also merged, you'll be able to send these events in beta.

If you've made these changes in InitialiseSettings.php, you'll need to schedule a Backport window deployment to sync out your config change to the production cluster. See Deployments and Backport windows for instructions.

Per-wiki configurations

The examples above (and many streams currently in production) are deployed to 'default' (all wikis), but what's important is that they're deployed to metawiki – because EventGate uses the stream configuration from meta.wikimedia.org for checking which streams are active and which schema titles they're associated with.

Suppose you have deployed an instrument to production and that instrument produces events to 'analytics.interwiki_link_hover', which is not yet configured. You want to "activate" this instrumentation on a smaller wiki (e.g. cawiki) first to gauge volume of events. Technically the instrumentation is already active, but because the stream it's producing events to is not configured yet those events will not be sent to the intake service.

Currently you cannot configure a stream exclusively for, say, cawiki. This is invalid:

'wgEventStreams' => [
	'cawiki' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover'
		],
	],
],

Well, it's valid but it doesn't do anything because that stream is not visible to EventGate. For remote activation/deactivation of instruments via stream configuration, there's two options for partial activation.

Approach 1: EventLogging registration

'wgEventStreams' => [
	'default' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover'
		],
	],
],

and

'wgEventLoggingStreamNames' => [
    'default' => [
        // NOT 'analytics.interwiki_link_hover'
    ],
	'cawiki' => [
		'analytics.interwiki_link_hover'
	],
],

This is the recommended approach, for performance reasons. An event produced to the 'analytics.interwiki_link_hover' stream will be outright rejected on all wikis except cawiki because that stream's configuration is only available to EventLogging on cawiki and nowhere else. There is no in-sample vs out-of-sample determination that needs to happen involving math and thus this approach is preferable for performance reasons.

Approach 2: Sampling and configuration override

The following is an alternative method to demonstrate sampling and configuration overriding, and is not recommended. It is included for demonstration only.

'wgEventStreams' => [
	'default' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover',
			'sampling' => [
			    'rate' => 0.0,
			]
		],
	],
	'cawiki' => [
	    [
	        'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover',
			'sampling' => [
			    'rate' => 1.0,
			]
	    ]
	]
],

and

'wgEventLoggingStreamNames' => [
    'default' => [
        // ...
        'analytics.interwiki_link_hover'
    ],
],

This will cause the 'analytics.interwiki_link_hover' stream to be visible to EventGate and deployed on all wikis' stream configurations with 0% sampling rate, except on cawiki where it will be configured to have 100% sampling rate.

The reason this approach is not recommended is because instead of the event being outright rejected like it would be in approach 1, the client has to perform (unnecessary in this case) calculations to determine whether the stream is in-sample or out-of-sample.

Viewing and querying events

In beta

TODO: set up a GUI consuming from EventStreams in beta that helps here.

In production

Events in production eventually make their way into the Analytics/Data_Lake where they are ingested into a Hive table in the event Hive database. From there they are queryable using Hive, Spark, or Presto, and also available for dashboarding in Superset via Presto.

The Hive table name will be a normalized version of the stream name. Our example stream's Hive table will be event.analytics_interwiki_link_hover.

Events are also available realtime by consuming them directly from Kafka. The stream name is prefixed with the source datacenter for the Kafka topic. WMF has two main datacenters: 'eqiad' and 'codfw'. To consume all events from Kafka for your stream, you should consume from both of these topics. Our example's Kafka topics will be 'eqiad.analytics.interwiki_link_hover and 'codfw.analytics.interwiki_link_hover'.

Viewing schema validation errors

Events that do not validate with their schema will not be produced. For events that are produced via EventLogging, validation error events will be produced into the stream eventgate-analytics-external.error.validation and will be ingested into Hive in the event.eventgate_analytics_external_error_validation table. These validation error events are also ingested into logstash and can be viewed in Kibana using this saved search.

Evolving

Let's say our 'analytics.interwiki_link_hover' stream is operating fine in production, and now we want to also log the link's text value. We'll need to add a new link_text field to the 'analytics/link_hover' schema.

Edit jsonschema/analytics/link_hover/current.yaml again and add the field and bump the version number in the $id. The new content of current.yaml should look like:

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.1.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link
      link_text:
        type: string
        description: text value of http anchor link
examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"analytics.interwiki_link_hover"},"event_dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging", "link_text": "EventLogging extension"}

Now, add and commit current.yaml again and jsonschema-tools will materialize new 1.1.0 version files.

$ git add jsonschema/analytics/link_hover/current.yaml
$ git commit -m 'analytics/link_hover - add link_text field and bump to version 1.1.0'
[2020-05-20 19:52:39.162 +0000]: Looking for modified current.yaml schema files in ./jsonschema/
[2020-05-20 19:52:39.229 +0000]: Materializing ./jsonschema/analytics/link_hover/current.yaml...
[2020-05-20 19:52:39.235 +0000]: Dereferencing schema with $id /analytics/link_hover/1.1.0 using schema base URIs ./jsonschema/,https://schema.wikimedia.org/repositories/primary/jsonschema/
[2020-05-20 19:52:39.264 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.1.0.yaml.
[2020-05-20 19:52:39.271 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.1.0.json.
[2020-05-20 19:52:39.275 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.yaml -> 1.1.0.yaml.
[2020-05-20 19:52:39.275 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.json -> 1.1.0.json.
[2020-05-20 19:52:39.276 +0000]: Created extensionless symlink ./jsonschema/analytics/link_hover/1.1.0 -> 1.1.0.yaml.
[2020-05-20 19:52:39.278 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest -> 1.1.0.yaml.
[2020-05-20 19:52:39.279 +0000]: New schema files have been materialized. Adding them to git: ./jsonschema/analytics/link_hover/1.1.0.yaml,./jsonschema/analytics/link_hover/latest.yaml,./jsonschema/analytics/link_hover/1.1.0,./jsonschema/analytics/link_hover/latest,./jsonschema/analytics/link_hover/1.1.0.json,./jsonschema/analytics/link_hover/latest.json
[analytics_link_hover 3c4f8c7] analytics/link_hover - add link_text field and bump to version 1.1.0
 7 files changed, 227 insertions(+), 4 deletions(-)
 create mode 120000 jsonschema/analytics/link_hover/1.1.0
 create mode 100644 jsonschema/analytics/link_hover/1.1.0.json
 create mode 100644 jsonschema/analytics/link_hover/1.1.0.yaml

New analytics/link_hover/1.1.0 files have been created, and the jsonschema/analytics/link_hover/latest symlinks have been updated to point to the 1.1.0 versions.

Now edit your instrumentation code to produce the event data with the link_text field and the udpated versioned $schema URI. Your instrumentation code should now look like this:

$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		// We've added link_text, which was added in schema verison 1.1.0, so we need to specify that this
		// event should be validated using schema verison 1.1.0.
		$schema: '/analytics/link_hover/1.1.0',
		link_href: link.href,
		link_title: link.title,
		link_text: link.text,
	}
	var streamName = 'analytics.interwiki_link_hover'
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Note that only backwards compatible changes are allowed. This means that the only type of change you can do to a schema is add new optional fields. jsonschema-tools will ensure that all schema versions are backwards compatible (as well as ensuring that the schema repository is in good shape). Jenkins will run CI tests when you push your schema change to gerrit, but you can also run the tests manually:

$ npm test
...
  Schema Compatibility in Repository ./jsonschema/
    analytics/link_hover
      Major Version 1
        ✓ 1.1.0 must be compatible with 1.0.0
...

Decommissioning

When deploying a new instrumentation event stream, you should also plan to decommission it one day. A schema should never be deleted, but all of the stream related code and configuration can be removed at anytime to stop producing an event stream. In the rare cases where the presence of the schema itself is problematic, we can delete it but it requires a coordinated effort to blacklist it and prevent alarms along the data pipeline.