You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Event Platform: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ottomata
No edit summary
 
imported>Ottomata
Line 5: Line 5:


* [[Event Platform/EventGate]] - Documentation about EventGate event intake services.
* [[Event Platform/EventGate]] - Documentation about EventGate event intake services.
* [[Event Platform/EventBus]] & [[Event Platform/EventBus/Administration]] - Documentation about EventBus mediawiki extension as well as legacy eventlogging-service-eventbus event intake service.
* [[Event Platform/EventStreams]] & [[Event Platform/EventStreams/Administration]] - Documentation about the public event publishing service.
* [[Event Platform/EventStreams]] & [[Event Platform/EventStreams/Administration]] - Documentation about the public event publishing service.
* [[Analytics/Systems/EventLogging]] & [[Analytics/Systems/EventLogging/Administration]] - Documentation about the original EventLogging Analytics event intake and distribution systems.
* [[Analytics/Systems/EventLogging]] & [[Analytics/Systems/EventLogging/Administration]] - Documentation about the original EventLogging Analytics event intake and distribution systems.
Line 30: Line 28:
In 2018, we started the Modern Event Platform program, which included EventBus's original analytics+production unification goal as well as other parts of WMF's event processing stack using open source (non-homegrown) components where possible.  The EventLogging python codebase was too WMF and Mediawiki specific to easily accomplish the unification, so it was decided to build a new more generic and extensible JSONSchema event service, eventually entitled EventGate.
In 2018, we started the Modern Event Platform program, which included EventBus's original analytics+production unification goal as well as other parts of WMF's event processing stack using open source (non-homegrown) components where possible.  The EventLogging python codebase was too WMF and Mediawiki specific to easily accomplish the unification, so it was decided to build a new more generic and extensible JSONSchema event service, eventually entitled EventGate.


In 2019, EventGate, along with other Modern Event Platform components, are intended to replace both eventlogging-service-eventbus, as well as the 'analytics' deployment of EventLogging services (e.g. eventlogging-processor).
In 2019, EventGate, along with other Modern Event Platform components, replaced eventlogging-service-eventbus, and is intended to eventually replace as the 'analytics' deployment of EventLogging services (e.g. eventlogging-processor).
[[Category:Services]]
[[Category:Services]]

Revision as of 19:04, 18 September 2019

The Wikimedia Event Platform refers to the various event stream distribution and processing systems we employ at the Wikimedia foundation. There are various components that make up the event platform, including 'legacy' and 'modern'. This page documents these different components and provides some historical context and also outlines future work and intentions.


More documentation can be found on component specific pages:

Platform Architecture Summary

TODO

Historical Overview

Some terms:

  • Event - A strongly typed and schemaed piece of data, usually representing something happening at a definite time. E.g. revision-create, user-button-click, page-load, etc.
  • Stream - A contiguous (often unending) collection of events (loosely) ordered by time.

The first 'event platform' at WMF was EventLogging. This system originally used ZeroMQ to transport messages between its various 'services' but was later improved to use Kafka. It was built for WMF product teams to be able to instrument and measure WMF features and usage on websites and apps. It used a (hardcoded) on-wiki event 'schema repository' to validate incoming events.

In 2015, an effort was made to extend the analytics focus of EventLogging to use for production events. This effort was dubbed 'EventBus' and culminated in three new components: the Mediawiki EventBus extension, the mediawiki/event-schemas git repository, and eventlogging-service-eventbus. eventlogging-service-eventbus is the first HTTP POST (internal) endpoint. It validated and produced events against more tightly controlled production schemas, and produced them to Kafka. EventBus was used to build the Change Propagation service. We originally intended to merge the analytics vs. production uses of EventLogging.

In 2018, we started the Modern Event Platform program, which included EventBus's original analytics+production unification goal as well as other parts of WMF's event processing stack using open source (non-homegrown) components where possible. The EventLogging python codebase was too WMF and Mediawiki specific to easily accomplish the unification, so it was decided to build a new more generic and extensible JSONSchema event service, eventually entitled EventGate.

In 2019, EventGate, along with other Modern Event Platform components, replaced eventlogging-service-eventbus, and is intended to eventually replace as the 'analytics' deployment of EventLogging services (e.g. eventlogging-processor).