Test Kitchen/Glossary

Platform components

Test Kitchen UI

Web application for creating and managing experiments and instruments, available at test-kitchen.wikimedia.org . Allows experimenters to define experiment configuration that is served to Experiment Enrolment Sampling Authorities. Formerly known as MPIC or xLab.

Automated analytics

System to define metric and then apply those definitions to query events from Hive and visualize them in Superset . See Test Kitchen/Automated analysis of experiments

SDK/Client Libraries

The methods provided by the Test Kitchen client libraries to submit events. Client libraries are available for JavaScript, PHP and Kotlin (in the future, Swift), enabling developers to create instrumentation in MediaWiki and mobile apps. The SDKs handle interaction with schemas and event streams, providing standardized methods for submitting events to EventGate. See Test Kitchen/SDK .

Instrumentation concepts

Contextual attributes

Fields in an event data object that provide information about the performer who triggered the event and the wiki where the event occurred. For example, the page title and page language. The values of contextual attributes that are included in the stream configuration are populated automatically by Test Kitchen when the event is generated. See Metrics Platform/Contextual attributes .

Event

A data object that represents a user interaction happening at a definite time. For example, a user clicking a button. Instruments work by sending event data objects to an event stream in response to user actions or app conditions.

Event stream

A continuous series of structured event data, as in Event Platform/EventStreams HTTP Service . An instrument publishes events to an event stream. Contextual attributes and analytics sampling are configured per event stream. To configure an event stream manually using MediaWiki's configuration files, see Test Kitchen/Stream configuration . Event data are stored in Hive .

Instrument

See mw:Product Analytics/Glossary#Instrument .

Instrumentation

See mw:Product Analytics/Glossary#Instrumentation .

L3SC review

A process in which new data collection activities are reviewed by Privacy Engineering and Privacy Legal depending on risk tier. See foundation:Legal:Data Collection Guidelines .

Data and schemas

Base schema

A reusable default schema provided by Test Kitchen to use with web or mobile app instruments. See Test Kitchen/Schemas .

Data contract

Refers to the Test Kitchen base schemas and the conventions for creating custom schemas. See Test Kitchen/Schemas .

Fragment

A partial schema designed to be combined with other fragments to build custom schemas. See Test Kitchen/Custom schemas .

Interaction data

Properties supported by the Test Kitchen schemas that can be customized to include data specific to an instrument. For example, action_context and element_id . See Test Kitchen/Schemas .

Monotable

Instruments using the same base schema can use the same event stream. The resulting data is available in a shared Hive table. This shared table and this architectural approach is sometimes referred to as the monotable. See Metrics Platform/Decision Records/Single Table Per Base Schema .

Schema

A data structure used by Test Kitchen to validate event data. Test Kitchen schemas as structured as JSON Schemas and written as YAML files. See Test Kitchen/Schemas .

Experimentation concepts

Bucketing

See #Experiment enrolment sampling .

Central Auth ID

A persistent identifier for logged-in users across wikis, used for enrolling subjects in logged-in experiments .

Edge unique ID

A browser cookie-based identifier used by Varnish for enrolling subjects in everyone experiments. Managed at the edge layer before requests reach MediaWiki. See https://meta.wikimedia.org/wiki/Edge_Uniques

Experiment

See mw:Product Analytics/Glossary#Experiment .

Experiment enrolment sampling

The process of enrolling users into experiments and consistently assigning an enrolled user a variant of the feature that is being experimented on. Also known as bucketing. In Test Kitchen, this is performed by Experiment Enrollment Sampling Authorities (Varnish for everyone experiments, MediaWiki for logged-in experiments). See Experiment enrolment sampling .

Experiment enrolment sampling authority

A component that performs experiment enrollment sampling for active experiments. Must have access to both (1) an identifier for the subject being enrolled and (2) a store of experiment configs. Test Kitchen uses two authorities: Varnish for "everyone" experiments (using edge unique IDs) and MediaWiki for logged-in experiments (using Central Auth IDs).

Experiment orchestration

See mw:Product Analytics/Glossary#Experiment orchestration .

Group

A specific treatment variant in an experiment (e.g., "control", "treatment_a", "treatment_b"). Each enrolled subject is assigned to exactly one group.

Hypothesis

See mw:Product Analytics/Glossary#Hypothesis .

Logged-in experiment

An experiment targeting only logged-in users. MediaWiki is the experiment enrolment sampling authority for logged-in experiments, using Central Auth ID to provide consistent experience across wikis.

Everyone experiment

An experiment targeting all users (both logged-in and logged-out). Varnish is the experiment enrolment sampling authority for everyone experiments, using edge unique IDs for per-device enrolment.

Variant

A version of a feature being tested in an experiment. In single-variant experiments, groups represent different values of one variable (e.g., button color). Multi-variant experiments test multiple variables simultaneously (e.g., button color AND background color).

Metrics and analysis

Analytics sampling

Analytics sampling is the logic that determines whether an event is included in data collection for an event stream, including the sampling unit and rate. See Analytics sampling .