Test Kitchen

Test Kitchen (previously Metrics Platform, Experimentation Lab, xLab ) is a platform that helps Wikimedia teams make data-driven decisions about product experiences.

Choose a workflow

Conduct an experiment

Run A/B tests to compare variants and measure impact of changes

Measure product health

Track metrics and user behaviour without testing variants

Is Test Kitchen right for you?

✓ Good fit if you're working on:

MediaWiki-based products (Wikipedia, mobile apps)
New features without existing Event Platform instrumentation
Products where you need standardized measurement

✗ Use Event Platform instead if you have:

Browser extensions, backend services outside of MediaWiki, or programming languages not covered by the SDK
Websites outside of MediaWiki (although these sites can still use the Test Kitchen schema conventions )

Guides

Using GrowthBook
Local development setup
Decommission an instrument
Regulation section
Measuring clickthrough rates
To be deprecated soon:
- Automated analysis of experiments
- Converting queries for product health monitoring

Templates

Experimentation templates

The Phabricator task templates below use [brackets] as notes and placeholders. Remember to remove/replace everything in [brackets].

Template	Suggested owner/driver	Usage notes
Experiment Plan Phab Form GDoc	Product Manager or Hypothesis Owner	Phab Form : The task is the source of truth for experiment details when planning, replacing the measurement plan standalone document. It should be a sub-task of the project/OKR hypothesis Epic, which might include other sub-epics related to design, community communications, research (for example). GDoc : You can use this as a working document for easier, faster collaboration, as an alternative to planning the experiment in Phabricator. Once the plan is finalized, either continue using this document as source of truth for experiment details or move the information to Phabricator, continue working there, and archive this document. Both are structured to follow the flow of creating an experiment in GrowthBook .
Experiment Instrumentation Specification GSheet	Product Analyst or Product Engineer	This document is the source of truth for all events which the experiment is expected to produce to enable calculation and analysis of its metrics in GrowthBook .
Experiment Implementation Phab Form	Product Engineer	This task should be a sub-task of the experiment Epic (see form above to create one) and is used to track implementation of the experiment, including instrumentation, QA, and configuration in GB.

Legacy templates:

Instrumentation specification for experiments (TK UI-based)
Experimentation scorecard (Draft)

Product health instrumentation templates

Measurement plan
Instrumentation specification (for non-experiment instrumentation)

Reference

Test Kitchen maintainer documentation

Maintainer introduction
Architecture
Troubleshooting
Test Kitchen UI
- Administration
- Test Kitchen UI Design Document
GrowthBook
- Administration
Documentation maintenance
Incident reports
SLOs

Support

If you have questions that are not covered by these pages, or additional guidance is needed, please do not hesitate to reach out using the Experiment Platform intake process .

Contributing

Contributions to the Test Kitchen client libraries are most welcome and appreciated. Learn more about contributing to Test Kitchen development

About

Built on the Event Platform , Test Kitchen provides standardized algorithms, behaviors, and basic necessities for web and app instrumentation, including:

a specialized Event Platform client designed to require less work in creating instrumentation
a predefined core interaction schema covering the most common data fields
a library of event schemas designed for use across a wide range of projects
a library of already existing instruments to be reused directly or as a starting point
easy means to build new schemas to further enrich events with contextual data
the ability to mix in different schemas depending on your needs
standardized session ID generation, consistent across MediaWiki, Android, and iOS
standardized session expiry
determining which events are in-sample or out-of-sample based on a specific identifier (currently: pageview, session, or app install ID).

To learn more about Test Kitchen and which components are available to use, see the FAQ .

Background

Previously, different teams implemented their own analytics solutions in isolation from one another. Those solutions were typically based on the Legacy EventLogging pipeline and, more recently, the Event Platform . Test Kitchen is an effort to unify that previous work and to establish uniformity and consistency across platforms. Its objectives include:

making it easier to implement and maintain instruments. Test Kitchen provides the SDKs and protocols that MediaWiki developers need to create sophisticated instruments in as few lines of code as possible while maintaining quality, rigor, and safety.
making it easier for analysts to support teams which are not their primary teams. Previously, every instrument has its own quirks and conventions, which need to be remembered by analyst that works with the data produced by the instrument – including analysts that are providing temporary support.
making it easier to leverage data from multiple platforms to yield insights into how our users use our whole ecosystem of products in unison.