Test Kitchen/Troubleshooting

This page provides troubleshooting instructions for Test Kitchen . It is relatively new so there haven't been many incidents to draw troubleshooting instructions from. Instead, this page provides a map of components of the system, guidance for finding experiment configuration details, and dashboards to review. Please add to the end of the page if you encounter incidents; these may help someone (or yourself) if similar things happen in the future.

Also, be aware of Test Kitchen/Test Kitchen UI/Administration for additional tips for operators conducting administration of the Test Kitchen UI.

A note about 14:30 UTC

Experiment configuration is generally set to go in force on a start date and end on an end date at 14:30 UTC (on both dates) for pre-planned experiments. When you see a start date or end date in Test Kitchen UI remember this. You can see a recurring "Test Kitchen UI" experiment window associated with Deployments for this purpose (heads up, daylight confusion time can cause confusion in the schedule, but the TK configuration API use 14:30 UTC).

It is possible for manual configuration to change a start date to be sooner, which can have the practical effect of "starting" an experiment (at least if the experiment is marked as On , which is what makes it eligible to be in effect in the first place, and we're in the middle of the start-end interval). So, when looking at logs, it's pretty typical for things to start showing up or ending around 14:30 UTC for experiments, but it's also possible for them to seemingly start at arbitrary times. Thus you may need to triangulate in the regular SAL , the analytics SAL , and other git history.

How do I…?

Review our SLOs?

Use the Test Kitchen SLO Grafana dashboard to review our SLOs. See also SLO/Test_Kitchen for the definition, documentation, and history of our SLOs.

See if an experiment is involved and causing problems?

When the TestKitchen extension determines experiments may be involved, it will add a context_ab_tests data structure to Logstash entries (as context.ab_tests in code) via the BeforePageDisplay hook called early in the page lifecycle; the context_ab_tests field may also have an empty value if no experiments are seemingly in play.

For example, see this Logstash dashboard snapshot where a filter is applied looking for the existence of context_ab_tests (you may want to apply a similar filter while using or building other dashboards)

Logstash record fields in an OpenSearch Dashboard showing that an A/B test is in force via Test Kitchen UI (formerly known as Experimentation Lab or xLab)

If you only want to see Logstash entries where a user was actually enrolled into an experiment (that is, according to the MediaWiki application server's enrollment routine in Test Kitchen extensionware), you likely want to set a custom filter to ensure the existence of context_ab_tests.enrolled .

If you are troubleshooting and see a correlation in Logstash with errors associated with specific A/B tests, you should take the following steps. We'll use an example here.

1. Identify where in code and configuration the the experiment may be triggered. This may be obvious from a stack trace, or it may not. Notice the value(s) of context_ab_tests.active_experiments and search in Codesearch . In these example screenshots you see a slug (machine readable name) of growthexperiments-get-started-notification . Take note of the enrolled property and the assigned property and the mapping with the pertinent experiment(s) involving Logstash entries and the corresponding code.

2. Access Test Kitchen UI (formerly known as Experimentation Lab / xLab / MPIC) and search for the slug.

3. Notice that the experiment is intended to be On and that it starts 25th September 2025. Click on the experiment hyperlink.

4. The User Identifier Type is mw-user , which means that inclusion in the experiment is based on sampling using the user ID for the wikis indicated in the Traffic section. Notice that the experiment runs through 15th December 2025 and that it is intended to run on five Wikipedias at 100% sampling. 100% sampling in this context means that when extension code (in this, GrowthExperiments) checks for the presence of this experiment, all logged in users can be considered for inclusion in the A/B test.

If the User Identifier Type is edge-unique , this means inclusion is based on sampling at the CDN edge ( Varnish ) via the edge unique cookie . The sampling rates at the CDN edge need to be much lower in order to keep the edge cache performant.

You can learn more about these "enrollment authorities" .

5. You'll notice there is a link in Test Kitchen to a Phabricator ticket where you can learn more information about the experiment. You can also view the list of experiments in Product and Technology to look for more documentation and who to potentially contact from a team conducting an experiment.

6. If you determine that the errors require action relatively soon, for example because of triggering noticeable buggy behavior, file a task on Phabricator and tag the task with the software development team and software component (usually an extension) executing the experiment, and then link to it on the #talk-to-experiment-platform channel on Slack. In your Slack message please include the task number and name of the task for findability, at-mentioning software developers involved with the experiment. Members of the Experiment Platform Team watch this channel regularly and can assist with troubleshooting with the software development team if necessary. If you don't have access to Slack, please feel free to reach out to members who are on #wikimedia-operations or #wikimedia-analytics on Libera IRC if you know their handles (including phued.x , cjmin.g , or dr0ptp4k.t whose bouncer usually catches and notify about messages).

7. If you determine that the errors are serious and require urgent treatment, follow the previous step, and please escalate by additionally at-mentioning engineering management and product management from both the software development team and Experiment Platform Team on Slack as well (DMs and adding people to the thread on #talk-to-experiment-platform are encouraged). If necessary, escalate further on #talk-to-data-engineering and #engineering-all on Slack with a pointer to your Slack message on #talk-to-experiment-platform .

8. Deactivation of an A/B test is an option in exceptional circumstances . Deactivation can invalidate assumptions about data collection for making important decisions about product features. Ideally you will reach agreement on Slack and an ad hoc Meet if this is necessary. If you're lucky you may find a way to roll forward with a hotfix instead.

However, if there is a threat of application data corruption or a site outage and you've been unable to reach people even through escalations, you may need to take measured action. Please state clearly on Slack and on the Phabricator task that you will do so. Wait a few minutes if you can, and then go back to the experiment in Test Kitchen, click on the three dot overflow button for the pertinent experiment, and choose Turn Off . When prompted, please carefully verify that you are turning off the correct experiment, then confirm your choice; then notify people that you have done so on Slack and on the Phabricator task.

Note: The Test Kitchen UI requires authentication via CAS-SSO requires a Wikimedia Developer Account and membership in the wmf or nda group.

It is sometimes easier to functionally deactivate an experiment through a Gerrit change. It's still important if going this route to over-communicate your intent to do so and to update people once you have done so.

View experiment configuration history over time?

With our IRC bot

Basic IRC logging of experiment configuration changes for experiments that are turned On or Off, or are in an On state but get a change (e.g., experiment end date), occurs on Libera IRC chat on the #wikimedia-analytics channel via the nickname wmftkbot . A continuous Toolforge tool job is run from user tk in order to poll the Test Kitchen API and log these messages.

You can find an archive of this logging activity at mw:Analytics/Server_Admin_Log and the Toolforge analytics SAL search .

The bot logs experiment related changes for all experiments. When it logs, it points out adds (experiments that are made to be On), removes (experiments that have passed their end date or that have been made to be Off), and the most notable changed fields (especially end date, but potentially - via manual intervention - also start date, sample rate, and variants/groups).

This bot runs on a continuous basis. It shows the polling round where it identified the change as (poll #) . Because the bot can restart and it's meant to be an application without persisted state, the poll value can reset to 1 if the bot is restarted for any reason. Typically (poll 1) will show all experiments that are indicated as On as of the time of that poll because the application doesn't attempt to reconstruct history; it is starting from a fresh slate.

Similarly, structural changes to Test Kitchen API response shape can result in bot logged messages. For example, on 7th November 2025 "mdot" domains (e.g., en. m. wikipedia.org) were removed from the TK API responses for edge-unique experiments, whereas previously they were included by default (to ensure uniform treatment for small form factors and large form factors alike). This resulted in a logged message. And as there was only one active edge-unique experiment with an On disposition whose end date hadn't passed, it was flagged as having a change.

wmftkbot https://toolsadmin.wikimedia.org/tools/id/tk 
!log Test Kitchen edge-unique experiments (poll 4689) - adds: none; removes: none; fields: fy2025-26-we3.1-image-browsing-ab-test, hcaptcha-on-french-wikipedia, xlab-mw-module-loaded-v2 - xLab/MPIC/TK tips at https://w.wiki/FwuD

For the curious, the mdot domains were removed from the API because they are no longer in use as of the Autumn (Northern Hemisphere) 2025, except for legacy purposes of redirects. Traffic normally now flows exclusively through what used to be termed "desktop" domains (e.g., en.wikipedia.org).

With the experiment / instrument journal

The Test Kitchen database can be queried directly for those with adequate production Kubernetes access to the dse-k8s cluster configuration and the Data Lake . A nicer UI in Test Kitchen is envisioned for interrogating the changes that have occurred to experiment configuration (it could provide colorized diffs, filters for type of change, etc.). In a nutshell, the following commands can be useful for querying the journal table. You'll see that the configuration column takes on a JSON shape.

stat1010.eqiad.wmnet $ mysql -h an-mariadb1001.eqiad.wmnet -u test_kitchen_production -p
mysql> USE test_kitchen_production;
DESCRIBE instruments_history
SELECT * FROM instruments_history WHERE <where clause here>\G;

Administer the IRC bot?

See Test Kitchen/Test Kitchen UI/Administration#Troubleshoot wmftkbot for instructions on how to administer the IRC bot.

Check whether Varnish is enrolling traffic into experiments?

Firstly, have you checked that there are active everyone experiments? You can do this by inspecting the Test Kitchen API response for the Varnish authority . If there are active everyone experiments, then you can check whether Varnish is enrolling traffic into experiments via the SRE Traffic Team / Edge uniques Grafana dashboard , which shows:

Edge Unique states
Edge Unique validation failures
The rate of requests enrolled into an experiment

If the graphs have changed significantly recently, then reach out to the SRE Traffic Team .

Check the health of the TestKitchen MediaWiki extension?

There are a couple of places to check the health of the TestKitchen MediaWiki extension:

The TestKitchen MediaWiki Extension Grafana dashboard , which shows:
1. The performance of the WANObjectCache and MainStash, which are used to store experiment and instrument configs for quick retrieval
2. The performance of Test Kitchen UI from the point of view of the MediaWiki application servers
The logs via the mediawiki OpenSearch dashboard

If the WANObjectCache hit rate and/or the MainStash miss rate have changed significantly, then reach out in the #engineering-all Slack channel or in the #wikimedia-operations Slack channel as the WAN cache hit rate has decreased.

Check the health of GrowthBook?

Currently, GrowthBook isn't configured to export metrics to Prometheus, which makes it impossible to monitor its health actively. T430118 [GrowthBook] Export metrics to Prometheus addresses this.

You can use the Test Kitchen - Service Overview Grafana dashboard to check the health of GrowthBook from the point of view of the Test Kitchen UI service.

Components

EventGate

Test Kitchen UI

Test Kitchen UI allows users to define and coordinate experiments. The CDN and Extension:TestKitchen fetch experiments via the Test Kitchen API every minute per cache node and per DC, respectively.

You can monitor the following:

The logs via the Test Kitchen UI production OpenSearch dashboard
The internal performance of Test Kitchen UI via the Service / Test Kitchen UI Grafana dashboard , which shows:
- The rate of all requests
- The rate of all errors
Directly via kubectl

If the rate of all requests is zero or unusually low or unusually high, then either the CDN or Extension:TestKitchen components are misconfigured or in an error state.

Real-world Examples

(please add examples here as you encounter them. This section will serve as a recipe book for solving or triaging quickly)

Discovering a Severe Data Loss Issue

FY25-26 SDS2.4.11 involved running an A/A experiment. Experiment Platform validated that events were being sent by the Test Kitchen SDKs and were arriving in the event.product_metrics_web_base Hive table. However, we didn't know what the rate of events arriving in the Hive table should be.

Part of signing off on SDS 2.4.11 was validating that EventGate didn't log any errors during the experiment. To do this, we checked:

The EventGate validation errors OpenSearch dashboard for validation errors for events on the product_metrics.web_base event stream for the duration of the experiment
The eventgate OpenSearch dashboard

As well as validation errors, we were interested in subject ID hoisting errors. EventGate will throw an HoistingError error when:

It can't parse the X-Experiment-Enrollment header from the CDN; or
It can't extract the subject ID from the X-Experiment-Enrollment header related to the experiment named in experiment.assigned event property

EventGate doesn't always include the event stream name as context in its logs. However, if an error is thrown, then information about the error is included in the err.* properties. So to find the number of subject ID hoisting errors logged during the experiment, we filtered the eventgate OpenSearch dashboard for err.name: HoistingError .

We found that EventGate was encountering a subject ID hoisting error for ~48.71% of events. We notified SRE Traffic and DPE immediately because the bug involved the CDN and EventGate. The bug was found to be in the X-Experiment-Enrollment header parser in EventGate. It was fixed very soon after it was reported.