Jump to content

This is a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Test Kitchen/Architecture

From Wikitech

This page provides a detailed overview of the Test Kitchen architecture and how its components work together to enable instrumentation and experimentation.

Test Kitchen is a distributed system that enables product teams to instrument features and run A/B tests on Wikipedia and other Wikimedia projects. The system is made of a large set of components that communicate by API or protocol in order to enable/disable experiments quickly, and collect and analyze experiment-related analytics events efficiently. This page will:

  1. Give an overview of the components owned and maintained by the Experiment Platform team and provide links to documentation for those components that aren't; and
  2. Step through the lifecycle of an experiment to show how those components interact with one another

Overview

A detailed overview of the architecture of Test Kitchen

Test Kitchen UI

The Experiment Platform team is working on replacing Test Kitchen UI with GrowthBook. You will not be able to configure experiments in Test Kitchen UI by the end of FY26/27 Q1.

Test Kitchen UI is the source of truth for experiment and instrument configuration for Varnish and MediaWiki. It combines information about instrument and experiment configurations from multiple sources into one API endpoint. Other components, like Varnish and the TestKitchen MediaWiki extension, fetch experiment and instruments configurations from it.

The two sources Test Kitchen UI combines information from are GrowthBook and its own local database. Instrument data is stored in the Test Kitchen local database. Experiment configuration settings are fetched from GrowthBook.

Instrument owners can configure and activate/deactivate their instruments in the UI, and the settings will be stored locally.

Experiment owners can configure and activate/deactive their experiments in GrowthBook. Previously, experiment configuration was also locally configured and stored in Test Kitchen UI. As of FY26/27 Q1 this is still supported, but GrowthBook is recommended. Local configuration of experiments will stop being supported in the upcoming quarter. Test Kitchen UI fetches and validates experiment configurations from GrowthBook and outputs them alongside the experiment and instrument configurations from Test Kitchen UI.

UI: https://test-kitchen.wikimedia.org

API:

Codebase: https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen

GrowthBook

GrowthBook is an open source feature flagging and experimentation software integrated into Test Kitchen. Currently, we are only using GrowthBook for experimentation.

Experiment settings are configured in GrowthBook, and experiment results can be viewed in GrowthBook.

Our installation of GrowthBook is entirely on-premises – GrowthBook queries our Data Lake using our Presto cluster and then analyzes the data in a statistics engine running on our servers.

Varnish

Varnish is the experiment enrollment sampling authority for cache-splitting and non-cache-splitting everyone experiments.

Varnish fetches the configurations of active everyone experiments every minute. It assigns the user agent an Edge Unique ID , enrolls the user agent in those experiments, and sends the experiment enrollment information to downstream components.

Varnish differs from the TestKitchen MediaWiki extension in that it persists the experiment configurations to disk. This is done for simplicity.

Codebases:

Dashboards:

TestKitchen MediaWiki extension

The TestKitchen MediaWiki extension has multiple responsibilities:

  • It acts as the experiment enrollment sampling authority for logged-in experiments
  • It fetches configurations for all active experiments to power the SDKs every minute
  • It fetches experiment enrollment information sent from Varnish and makes it available to the PHP SDK
  • It hosts the PHP and JavaScript SDKs and make them available to other MediaWiki code

The TestKitchen MediaWiki extension caches the experiment and instrument configurations in two places:

  1. The WANObjectCache (10 minutes)
  2. The MainStash (indefinitely)

This caching strategy prioritizes site performance over freshness as experiment and instrument configurations are fetched on every request served by the MediaWiki application servers. In the event that both places don't have experiment or instrument configurations, then all experiments and instruments are considered disabled.

Codebase: https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/TestKitchen,general

EventBus MediaWiki extension

The TestKitchen PHP SDK (provided by the TestKitchen MediaWiki extension) uses the EventBus MediaWiki extension to send analytics events to EventGate.

EventGate

EventGate, the Event Platform event intake service, receives, validates, and routes analytics events from experiments and instruments to the Data Lake .

Superset

A general-purpose analysis platform where where experiment results for experiments configured in Test Kitchen UI can be visualized. Results for experiments configured in GrowthBook are viewed in GrowthBook.

The experiment configurations are fetched from the Test Kitchen UI API every day by the load_configurations_daily Airflow job . The experiment results are calculated every hour by the analyze_experiments_hourly Airflow job .

Entrypoints

UI: https://superset.wikimedia.org/superset/dashboard/experiment-analytics/

Airflow: https://airflow-analytics-product.wikimedia.org/home?tags=test_kitchen

Codebases:

The lifecycle of an experiment

This section will change once GrowthBook is the source of truth for experiments

The Test Kitchen/Conduct an experiment guide details all of the phases in the lifecycle of an experiment. This section focuses on the Launch and Evaluate phases to clarify how all of the components outlined above interact with one another.

Launch

When an experiment is started in GrowthBook:

  1. ~1 minute after the experiment is started Test Kitchen UI fetches and validates the experiment configuration
    1. If the experiment configuration is not valid, Test Kitchen UI:
      1. Sets the experiment status in GrowthBook to draft
      2. Posts a comment on the Phabricator task in the phabricator_url field
    2. If the experiment configuration is valid, Test Kitchen UI:
      1. Persists the experiment configuration to the database
      2. Adds the experiment configuration to the Test Kitchen APIs

When a cache-splitting everyone experiment is activated in Test Kitchen UI :

  1. The experiment configuration is present in the Test Kitchen APIs
  2. The experiment traffic allocation and groups are converted to ranges between 1 and 100,000 for Varnish
    1. For example: An experiment with a 1% traffic allocation, and with control and treatment groups, are converted to control: [0,999) and treatment: [50000, 50999) , respectively
  3. ~1 minute after the experiment is activated:
    1. Varnish fetches the experiment configuration and persists it to disk
    2. MediaWiki fetches the experiment configuration and stashes it in the MainStash for a week
  4. The automated analysis pipeline fetches the experiment configuration and persists it to Hive

When Varnish processes a request:

  1. It checks that request has an Edge Unique cookie
  2. If the request doesn't have an Edge Unique cookie, then it initializes one and sets it
    1. Note: Edge Unique cookies are set with the secure and httponly attributes and so cannot be read by JavaScript
  3. For each everyone experiment:
    1. It hashes the Edge Unique with the name of the experiment
    2. It converts the hash to a number in the range [1, 100,000)
    3. It checks if the number falls in one of the ranges
    4. If the number falls in a range, then it stores the name of the experiment and group name associated with the range
  4. It converts the set of experiment, group tuples to a backend request header, X-Experiment-Enrollments
    1. For example: X-Experiment-Enrollments: foo=bar;baz=qux;
  5. It checks whether there is a response with a matching X-Experiment-Enrollments header set
  6. If there is no response with a matching X-Experiment-Enrollments header set, then it sends the request to the backend
  7. If the backend response is cacheable, it ensures that backend response is cached with the X-Experiment-Enrollments header set
  8. It sets a Server-Timing header to the value of the X-Experiment-Enrollments header
    1. For example: Server-Timing: WMF-Uniq;desc="foo=bar;baz=qux;"
  9. It sends the response to the user agent

When the MediaWiki app servers receive the request, the MediaWiki TestKitchen extension is loaded and the the BeforeInitialize hook fires:

  1. It parses and validates the X-Experiment-Enrollments header into a set of experiment, group tuples
  2. It checks if the user is logged in and has a central ID
  3. It fetches all experiment configurations
    1. It fetches all experiment configurations from the WANObjectCache
    2. If no experiment configurations were returned, then it fetches all experiment configurations from the MainStash
      1. If no experiment configurations were returned, then it stores the empty list in the WANObjectCache for a minute
      2. It experiment configurations were returned, then it stores the experiment configurations in the WANObjectCache for 10 minutes
  4. If the user is logged and has a central ID, then for each logged-in experiment:
    1. It hashes the Edge Unique with the name of the experiment and converts it to a number in the range [0, 1)
    2. It converts the experiment traffic allocation to a number in the range [0, 1)
    3. If the number in (1) is less than the number in (2), then:
      1. It uses the number in (1) to select a group name
      2. It stores the name of the experiment and group name
  5. It combines the set of experiment, group tuples in (1) and (4)
  6. It initializes the PHP and JavaScript SDKs with the results of (5) and (3)
  7. It adds the ext.testKitchen ResourceLoader module to the response

When the ext.testKitchen ResourceLoader module loads:

  1. It parses and validates the Server-Timing header sent by Varnish into a set of experiment, group tuples
  2. It parses and validates the mpo query parameter into a set of experiment, group tuples
  3. It combines the set of experiment, group tuples sent by MediaWiki with (1) and (2)
  4. It defines the mw.testKitchen namespace and methods

When the feature code requests details of an experiment via the mw.testKitchen.getExperiment() method:

  1. It looks up the group name and experiment configuration
  2. It initializes an instance of the Experiment class with the experiment name, group name, and experiment configuration

When the feature code sends an experiment-related analytics event using the Experiment#send() method :

  1. It constructs a valid Event Platform event
  2. It merges the given action and interaction data into the event
  3. It merges the given names of contextual attributes with the names of contextual attributes from the experiment configuration
  4. It sets the named contextual attributes
  5. It initializes the experiment object with the experiment configuration and adds it to the event
    1. NOTE: For everyone experiments, the value of experiment.subject_id is always "awaiting"
  6. It queues the event for sending after 5 seconds, or when the user hides the page or navigates away from it
  7. It sends the event to the global experiment-related analytics event intake URL, /evt-103e/v2/events

When Varnish processes a request to send an event:

  1. It checks that request has an Edge Unique cookie that has been sent to the user agent and received by it at least once
  2. For each everyone experiment:
    1. It hashes the Edge Unique with the name of the experiment
    2. It converts the hash to a number in the range [1, 100,000)
    3. It checks if the number falls in one of the ranges
    4. If the number falls in one of the ranges, then it stores the name of the experiment, the hash, and group name associated with the range
  3. It checks that the set of experiments, hashes, and group names has at least one entry
  4. It converts the set of experiment, group tuples to a backend request header, X-Experiment-Enrollments
    1. For example: X-Experiment-Enrollments: foo=bar/foo_hash;baz=qux/baz_hash;
  5. It sends the request to the backend
  6. It responds with a synthetic HTTP 204 No Content response

When EventGate receives an event:

  1. It validates it against the /analytics/product_metrics/web/base/2.0.0 schema
  2. It validates and parses the X-Experiment-Enrollments header into a set of experiment, hash, group name tuples
  3. It finds the tuple that matches { experiment.enrolled, *, experiment.assigned }
    1. If no tuple is found, then an error is thrown and the event is sent to the eventgate-analytics-external.error.validation stream
  4. It sets the experiment.subject_id to the hash from the tuple
  5. It adds the event to the eqiad.product_metrics.web_base and codfw.product_metrics.web_base Kafka topics
  6. ~3 hours after the event is added to the Kafka topics, Gobblin ingests the event into the Data Lake

Evaluate

When the growthbook_hourly DAG runs:

  1. It fetches experiment events from from the event.product_metrics_web_base Hive table where:
    1. experiment.coordinator is default
  2. It inserts all experiment events into the wmf_experiments.experiment_event_v1 timestamp-partitioned Iceberg table
  3. It fetches the experiment assignment events in the wmf_experiments.experiment_event_v1 timestamp-partitioned Iceberg table and
    1. Computes the user_auth_status and user_platform dimensions for each event
    2. Inserts the events into the wmf_experiments.experiment_assignment_v1 table

GrowthBook queries both the experiment_event_v1 and wmf_experiments.experiment_assignment_v1 tables using our Presto cluster when computing experiment results either periodically or when request by the user.

Legacy

The Experiment Platform team is working on replacing this MVP system with GrowthBook, which is how all future experiments conducted with Test Kitchen will be analyzed. This system will be decommissioned by the end of FY26/27 Q1.

When the automated analysis pipeline runs:

  1. It fetches experiment configurations from Hive
  2. For each experiment:
    1. For each metric:
      1. It computes the value of the metric from the experiment-related analytics events that have been received
      2. It persists the computed value to disk

See also

Retrieved from " https://wikitech.wikimedia.org/w/index.php?title=Test_Kitchen/Architecture&oldid=2445215 "