Changeprop

changeprop (or Change Propagation ) is the name given to a service that processes change events generated by MediaWiki and stored in Kafka. Various actions are taken based on the messages read from Kafka . Common actions take the form of HTTP requests or CDN purges. changeprop-jobqueue is an instance of changeprop which works on MediaWiki JobQueue jobs instead of the rules which are processed by the regular changeprop instance.

What it does

Changeprop uses Kafka to ensure guaranteed delivery. We use the Apache Kafka message broker to attain at least once delivery semantics: once an event is in Kafka, we can be sure that it will be processed and an event will follow. This allows us to build very long and complex sequences of dependencies without fear of loss of events.
Automatic retries with exponential delays, large job deduplication, and persistent error tracking via a dedicated error topic in Kafka
The config system allows us to add simple update rules with only a few lines of YAML and without code changes or deploys
Fine-granted monitoring dashboard allows us to track rates and delays for individual topics, rates of event production and much more. Changeprop graphs can occasionally be used to discover bugs in other parts of the infrastructure around it.
The main changeprop service creates events in order to react to changes on-wiki, refreshing caches, regenerating pages in response to content elsewhere.
The changeprop-jobqueue service processes jobs created by Mediawiki and other services. These jobs are POSTed to and then executed by hosts in the mw-jobrunner Mediawiki cluster.

How it works

Changeprop reads events from Kafka . The topics changeprop reads from are defined in config.yaml - the dc_name variable is a prefix to the topic defined on a per-rule basis. So for example in eqiad for the mw_purge rule which uses the resource_change topic, the full topic will be eqiad.resource_change . Each rule specifies the topic to which it subscribes.

Rules

Rules define a list of cases to which a rule is to respond. General rule properties allow the definition of things like retries, delays and other features.

The "match" section of a rule dictates a pattern to match, which can include URL matching and tag matching (for example, mw_purge events also contain "tags":["purge"] and will only match if the URL pattern and the URL matches the pattern specified). URL match patterns are frequently used to target specific sites (for example have a rule only apply to Wiktionary) or classes of article. Matches can also be fine tuned to not match using not_match. If the match it satisfied, the exec section is executed. The exec will generally be a HTTP request of a defined method to the specified URI. A rule can have multiple match and corresponding exec sections in its cases list - if a pattern is created where matches are mutually exclusive, a rule can act as a switch statement using the same topic and the same semantics but different matches. Headers and other parameters can be defined for an exec section - see the existing rules for details .

Service interactions

Changeprop talks to Redis to manage rate limiting and exclusion lists for problematic or high-traffic articles. All communication is done via Nutcracker . In Kubernetes, a local Nutcracker sidecar container runs within the changeprop pod, proxying access to a list of redis servers.

Many of changeprop's operations are accomplished by sending HTTP requests to the Page Content Service. .

What the rules do

This is a general overview of the rules that are explicitly processed by the changeprop pods running in Kubernetes. This is not an exhaustive list and is only a reference - refer to the helm configuration and dashboards for a view of current jobs.

PCS pre-generation

These rules warm the mobileapps/Page Content Service endpoints summary , mobile-html and media-list endpoints. See mw:Page_Content_Service for more details.

Rule	Topic	What it does
`pcs_rerender_native_on_edit`	`mediawiki.revision-create`	Rerender PCS content when an edit changes things.
`pcs_rerender_mobile_html_native_transcludes`	`change-prop.transcludes.resource-change`	Re-renders pages that transclude changed content. Pages with namespace File, Talk, Template and Discussion are ignored.
`pcs_rerender_mobile_html_native_wikidata_change`	`change-prop.wikidata.resource-change`	Re-renders containing changed wikidata content
`pcs_rerender_native_on_null`	`resource_change`	Handles generic resource-change purges such as null edits
`pcs_rerender_native_on_visibility_change`	`mediawiki.revision-visibility-change`	Re-renders both the specific revision and the current page, which is useful for cleaning caches
`pcs_rerender_native_on_page_move`	`mediawiki.page-move`	Re-renders new and old pages on a page move
`pcs_rerender_native_on_page_delete`	`mediawiki.page-delete` , `mediawiki.page-suppress`	Clean PCS content for deleted pages
`pcs_rerender_native_page_images_summary`	`mediawiki.page-properties-change`	Refresh the summary endpoint when a page image is changed

CDN purging

Rule	Topic	What it does
`generate_purge_varnish`	`resource_change`	Only operates on `/api/rest_v1/` URIs. Emits a `resource_change` event to trigger purges.
`generate_purge_varnish_transcludes`	`resource_change`	The same as above but for transclusions

Dependency and link updates

Rule	Topic	What it does
`page_edit`	`mediawiki.revision-create`	Upon a changing edit, checks for what pages transclude the edited page and creates further transclusion events.
`on_transclusion_update`	`change-prop.transcludes.resource-change`	Fetches further batches of transclusion events
`page_create`	`mediawiki.page-create`	Updates backlinks so red links to a new page turn blue
`page_delete`	`mediawiki.page-delete` , `mediawiki.page-suppress`	Updates backlinks to red on a page being deleted.
`page_restore`	`mediawiki.page-undelete`	Updates backlinks to blue on a page being undeleted
`on_backlinks_update`	`change-prop.backlinks.resource-change`	Rule to continue paging through the rest of a page's backlinks

Wikidata descriptions

Rule	Topic	What it does
`wikidata_description_on_edit`	`mediawiki.revision-create`	Rerender wiki pages that use a wikidata entity when a description is changed
`wikidata_description_on_undelete`	`mediawiki.revision-create`	Same as above but for undeletes

Lift Wing model scoring

All Lift Wing rules consume mediawiki.page_change.v1 and POST to inference.discovery.wmnet:30443 , which in turn self-publishes to Eventgate.

Rule	Scope	What it scores
`liftwing_revertrisk-language-agnostic`	All wikis, except commons, meta, sources, species and wikidata	Probability that an edit will be reverted
`liftwing_revertrisk-multilingual`	A specific list of wikis, see config	Larger multilingual revert-risk model
`liftwing_revertrisk-wikidata`	Wikidata only	Probability that a wikidata edit will be reverted
`liftwing_drafttopic`	enwiki edits and creates	Topic classification for drafts
`liftwing_outlink-topic-model`	Most wikis	Article topic prediction from outgoing links
`liftwing_article-country`	Most wikis	Geographical associations for an article
`liftwing_revise-tone-task-generator`	A small list of wikis including enwiki	Generates tone related suggested edits

What the jobs do

This is a general overview of the rules that are explicitly processed by the changeprop-jobqueue pods running in Kubernetes. Similar to the rules list, this is a reference and not exhaustive documentation of all jobs.

High-traffic jobs

Each job listed in high_traffic_jobs_config gets its own rule, and therefore its own dedicated consumer, so it cannot be blocked by (or block) other jobs.

Job	What it does
`categoryMembershipChange`	Writes recentchanges entries when an edit adds or removes a page from categories
`CategoryCountUpdateJob`	Updates member counts on category pages
`cdnPurge`	Issues CDN/Varnish purges
`ORESFetchScoreJob`	Fetches and caches ML revision scores for new edits
`RecordLintJob`	Records Linter errors reproted by Parsoid
`wikibase-addUsagesForPage`	Records which Wikidata entities on-wiki pages use.
`constraintsRunCheck`	Runs Wikidata property constraint checks on an edited entity and caches the results
`fetchGoogleCloudVisionAnnotations`	Fetches image labels from Google Cloud Vision for Commons files
`notificationGetStartedJob`	Sends "get started" nudge to new accounts
`notificationKeepGoingJob`	Sends "keep going" nudge
`notificationReEngageJob`	Sends engagement nudge
`newcomerTasksCacheRefreshJob`	Refreshes a newcomer's cached suggested-edits task list
`refreshUserImpactJob`	Recomputes a user's impact stats (edits, pageviews)
`processMediaModeration`	Submits Commons media to the external PhotoDNA service - maybe unused?
`LocalGlobalUserPageCacheUpdateJob`	Refreshes local cached copies of a global user page
`UpdateTranslatablePageJob`	Updates translation units after a page is marked for translation
`RenderTranslationPageJob`	Re-renders translated pages upon change
`DispatchChanges`	Dispatches wikibase entity-change notifications to subscribed client wikis
`EntityChangeNotification`	Turns a wikibase entity change into local page updates
`wikibase-InjectRCRecords`	Creates recentchange events for Wikidata changes impacting pages
`parsoidCachePrewarm`	Pre-generates Parsoid HTML for a new revision so readers and the API do not pay the first-render cost

Partitioned jobs

Partitioned jobs are fanned out onto cpjobqueue.partitioned.* topics, which map to a specific MySQL section .

Job	What it does
`refreshLinks`	Reparses pages and rebuilds link, template and category tables after a template or entity change
`htmlCacheUpdate`	Invalidates page caches for pages affected by a change

Latency-sensitive jobs

Latency sensitive jobs are low volume jobs where fast execution is important

Job	What it does
`AssembleUploadChunks`	Stitches a chunked upload back into a single file
`PublishStashedFile`	Publishes a stashed async upload into the file repo
`UploadFromUrl`	performs server-side fetch for upload-by-URL

All other jobs

The changeprop-jobqueue configuration has a concept of low_traffic_jobs using the glob ^mediawiki.job.* , while filtering out all jobs with explicit rules and excluded_jobs (for example webVideoTranscodePrioritized , which is handled by Mercurius ). This ensures that all jobs not explicitly assigned concurrencies/consumers will still get processed, but at a less guaranteed rate.

Where it runs

Changeprop currently runs in Kubernetes in codfw and eqiad. There is also an instance in the staging cluster that does not process prod traffic. In labs, changeprop runs in regular Docker on deployment-changeprop-1.deployment-prep.eqiad1.wikimedia.cloud.

Adding features

Adding a new rule

Add the rule to deployment-charts/charts/changeprop/templates/_config.yaml
Bump the Chart.yaml version
Commit, get review and merge
Deploy changeprop and changeprop-jobqueue from the deployment host using Kubernetes/Deployments#Code_deployment/configuration_changes

Deploying

To Kubernetes

Changeprop uses the Kubernetes/Deployments workflow to deploy changes.

To deployment-prep

In the Beta Cluster, Changeprop runs in Docker on deployment-changeprop-1.deployment-prep.eqiad1.wikimedia.cloud. The configuration passed to changeprop is generated by scripts in the deployment-charts repository, in order to use the same templates and avoid deviation. This means that if you want to change the configuration in beta/deployment-prep, you will first need to edit the configuration in deployment-charts . The values for deployment-prep are stored in the values-beta-changeprop.yaml file.

Generating the configuration

In deployment-charts , cd to charts/changeprop and run make_beta_config.py . The output from this command will be the configuration to be deployed.

For example, to generate the changeprop configuration from your localhost:

$ git clone https://gerrit.wikimedia.org/r/operations/deployment-charts
$ cd deployment-charts/charts/changeprop
$ python3 -m venv venv
$ venv/bin/pip3 install pyyaml
$ venv/bin/python3 make_beta_config.py . changeprop

To generate the jobqueue configuration:

$ git clone https://gerrit.wikimedia.org/r/operations/deployment-charts
$ cd deployment-charts/charts/changeprop
$ python3 -m venv venv
$ venv/bin/pip3 install pyyaml
$ venv/bin/python3 make_beta_config.py . changeprop

Deploying the configuration

The configuration is in config.yaml in a docker volume on deployment-changeprop-1.deployment-prep.eqiad1.wikimedia.cloud and deployment-docker-cpjobqueue01.deployment-prep.eqiad.wmflabs, named changeprop and cpjobqueue respectively. Configuration needs to be edited within this volume. The host directory can be discovered using `docker volume inspect`.

Ensure that the config is world readable when copying in a new file. Then run service changeprop restart to load the configuration. Files other than config.yaml in this volume will be ignored.

For example, to generate and deploy the changeprop configuration from your localhost:

$ git clone https://gerrit.wikimedia.org/r/operations/deployment-charts
$ cd deployment-charts/charts/changeprop
$ python3 -m venv venv
$ venv/bin/pip3 install pyyaml
$ venv/bin/python3 make_beta_config.py . changeprop |
  ssh deployment-changeprop-1.deployment-prep.eqiad1.wikimedia.cloud \
    sudo sh -xc \''cat > $(docker volume inspect changeprop -f {{.Mountpoint}})/config.yaml && systemctl restart changeprop'\'

To generate and deploy the cpjobqueue configuration:

$ git clone https://gerrit.wikimedia.org/r/operations/deployment-charts
$ cd deployment-charts/charts/changeprop
$ python3 -m venv venv
$ venv/bin/pip3 install pyyaml
$ venv/bin/python3 make_beta_config.py . jobqueue |
  ssh deployment-changeprop-1.deployment-prep.eqiad1.wikimedia.cloud \
    sudo sh -xc \''cat > $(docker volume inspect cpjobqueue -f {{.Mountpoint}})/config.yaml && systemctl restart cpjobqueue'\'

Ideally the docker volume would have been pre-created with a fixed host path.

Testing

changeprop can be tested by issuing events to Kafka that changeprop will consume. An example test command against the resource_change topic for the k8s staging cluster is: cat mw_purge_example.json | kafkacat -b localhost:9092 -p 0 -t 'staging.resource_change' .

All IDs in these examples are random UUIDs. Not varying UUID between tests runs the risk of being seen as a duplicate event and being skipped. The "dt" field should also be changed to be close to the current time and date, as changeprop will not take action on older events.

mw_purge

{
  "$schema": "/resource_change/1.0.0",
  "meta": {
    "domain": "en.wikipedia.org",
    "dt": "2020-04-02T17:16:25Z",
    "id": "22350141-bbe2-488d-9f73-a1aa6094ac5c",
    "stream": "resource_change",
    "uri": "https://en.wikipedia.org/wiki/Draft:Editta_Braun"
  },
  "tags": [
    "purge"
  ]
}

null_edit

{
  "$schema": "/resource_change/1.0.0",
  "meta": {
    "domain": "fr.wikipedia.org",
    "dt": "2020-04-02T17:16:28Z",
    "id": "b92d40b0-3206-469d9615-2fbf61a04418",
    "stream": "resource_change",
    "uri": "https://fr.wikipedia.org/wiki/Oribiky"
  },
  "tags": [
    "null_edit"
  ]
}

How to monitor it

There is a Grafana dashboard for Changeprop . The various graphs provide information about things such as rule execution rate and rule backlogs for each rule for various streams.

Rule backlog is the time between the creation of event and the beginning of processing. If the backlog grows over time - change propagation can't keep up with the event rate and either concurrency should be increased, or some other action taken. Backlogs can have occasional spikes, but steady backlog growth is a clear indication of a problem.

Debugging

Querying configuration

Changeprop's configuration can be queried if you have access to deploy1001:

ssh to the deploy server for the datacenter
cd to the appropriate directory (for example /srv/deployment-charts/helmfile.d/services/staging/changeprop)
run kube_env changeprop $CLUSTER to set up your Kubernetes environment
show the configuration via kubectl describe configmap changeprop-staging-base-config

The suffixes nutcracker-config and metrics-config are also available as configmaps.

Non-issues

Periodically Changeprop will log a message along the lines of the following:

{"name":"change-propagation","hostname":"changeprop-staging-684b9ddbd-4wdkn","pid":141,"level":"ERROR","err":{"message":"Local: Broker transport failure","name":"changeprop-staging","stack":"Error: Local: Broker transport failure\n    at Function.createLibrdkafkaError [as create] (/srv/service/node_modules/node-rdkafka/lib/error.js:334:10)\n    at /srv/service/node_modules/node-rdkafka/lib/kafka-consumer.js:448:29","code":-195,"errno":-195,"origin":"kafka","rule_name":"page_create","executor":"RuleExecutor","levelPath":"error/consumer"},"msg":"Local: Broker transport failure","time":"2020-04-29T13:10:17.443Z","v":0}

This can be ignored as long as the occurrences aren't too close together (currently they happen roughly once every hour in staging), they will not interrupt normal operation of changeprop.

Where it lives

Changeprop's code can be cloned from Gerrit at ssh://gerrit.wikimedia.org:29418/mediawiki/services/change-propagation . It can be browsed in Phabricator .
Changeprop is deployed to Kubernetes as a Helm chart. It lives in the deployment-charts repo.
The example config.yaml file contains many illuminating examples of how rules are matched and processed.
There is a per-environment Helmfile values file which overrides the defaults configured in the Helm chart's values file. This is the file for staging values, there are corresponding production files in the per-DC directories.