You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Event*: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ottomata
 
imported>Ottomata
Line 16: Line 16:


=== EventLogging Extension ===
=== EventLogging Extension ===
The [https://www.mediawiki.org/wiki/Extension:EventLogging EventLogging Mediawiki extension] is used by Mediawiki to send JSON data to EventLogging by sending a <tt>GET</tt> request with query parameter encoded JSON data to a special Varnish endpoint.  Requests to this Varnish endpoint are logged (via varnishkafka), and make there way into downstream EventLogging services.
The [https://www.mediawiki.org/wiki/Extension:EventLogging EventLogging Mediawiki extension] is used by Mediawiki to send JSON data to EventLogging by sending a <tt>GET</tt> request with query parameter encoded JSON data to a special Varnish endpoint.  Requests to this Varnish endpoint are logged (via varnishkafka), and make their way into downstream EventLogging services.


== [[EventBus]] ==
== [[EventBus]] ==
Line 31: Line 31:
== EventStreams ==
== EventStreams ==


[https://github.com/wikimedia/mediawiki-services-eventstreams EventStreams] is a public facing service.  It exposes streams of events via HTTP [https://en.wikipedia.org/wiki/Server-sent_events SSE/EventSource].  This service replaces the Mediawiki RecentChange specific service [[RCStream]], and perhaps eventually will also deprecate [https://www.mediawiki.org/wiki/API:Recent_changes_stream#How_it_works_on_Wikimedia_wikis irc.wikimedia.org].
[[EventStreams]] is a public facing service.  It exposes streams of events via HTTP [https://en.wikipedia.org/wiki/Server-sent_events SSE/EventSource].  This service replaces the Mediawiki RecentChange specific service [[RCStream]], and perhaps eventually will also deprecate [https://www.mediawiki.org/wiki/API:Recent_changes_stream#How_it_works_on_Wikimedia_wikis irc.wikimedia.org].

Revision as of 16:30, 14 February 2017

EventBlabla what?! So Many Event* services. This page disambiguates, and links to their more extensive documentation.

In all of the following, an 'event' is defined as a structured and schema-ed data object. The data object is usually serialized as JSON with the schema specified in and validated with JSONSchema. WMF does use some Avro based events, but none of these pass through any of the Event* services described below.

EventLogging

EventLogging is the original event framework used by WMF. It consists of 3 main parts.

EventLogging Services

The EventLogging python codebase consists of various services that can be mixed and matched to consume, validate, transform, and produce streams of JSON events between different endpoints. EventLogging was originally created for Mediawiki analytics purposes, but is now also used to support production streams of events via an HTTP eventlogging-service used for the EventBus service.

EventLogging Schemas

For the original analytics EventLogging use case, all schemas are maintained and stored on meta wiki. The default EventLogging validation logic looks for matching schemas in this online http/wiki repository.

EventLogging Extension

The EventLogging Mediawiki extension is used by Mediawiki to send JSON data to EventLogging by sending a GET request with query parameter encoded JSON data to a special Varnish endpoint. Requests to this Varnish endpoint are logged (via varnishkafka), and make their way into downstream EventLogging services.

EventBus

Like EventLogging, 'EventBus' refers to a few different pieces of infrastructure. More detail can be found on the EventBus documentation page.

eventlogging-service-eventbus

This is a particular deployment of eventlogging-service that is used to accept internally produced events over HTTP. It is configured to use JSONSchemas from the mediawiki/event-schemas repository to validate incoming events.

Event Schemas Repository

This repository is the main source of truth for production event schemas. All new production events should be schemaed, and should exist in this repository. eventlogging-service-eventbus validates incoming events against schemas in this repository.

EventStreams

EventStreams is a public facing service. It exposes streams of events via HTTP SSE/EventSource. This service replaces the Mediawiki RecentChange specific service RCStream, and perhaps eventually will also deprecate irc.wikimedia.org.