You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Changeprop

From Wikitech-static
Revision as of 17:55, 2 April 2020 by imported>Alex Paskulin (→‎Testing: Added syntaxhighlight)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

changeprop (or Change Propagation) is the name given to a service that processes change events generated by Mediawiki and stored in Kafka. Various actions are taken based on the messages read from Kafka.

Where it is

Changeprop currently runs in the scb cluster and in Kubernetes in a currently limited capacity with plans to move fully in future.

How it works

Changeprop reads events from Kafka. The topics changeprop reads from are defined in config.yaml - the dc_name variable is a prefix to the topic defined on a per-rule basis. So for example in eqiad for the mw_purge rule which uses the resource_change topic, the full topic will be eqiad.resource_change. Each rule specifies the topic to which it subscribes.

Rules define a list of cases to which a rule is to respond. The "match" section of a rule dictates a pattern to match, which can include URL matching and tag matching (for example, mw_purge events also contain "tags":["purge"] and will only match if the URL pattern and the URL matches the pattern specified). Matches can also be fine tuned to not match using not_match. If the match it satisfied, the exec section is executed. The exec will generally be a HTTP request of a defined method to the specified URI. A rule can have multiple match and corresponding exec sections in its cases list. Headers and other parameters can be defined for an exec section - see the existing rules for details.

Changeprop talks to Redis to manage rate limiting and exclusion lists for problematic or high-traffic articles. All communication is done via Nutcracker. In Kubernetes, a local Nutcracker sidecar container runs within the changeprop pod, proxying access to a list of redis servers.

Many of changeprop's operations are accomplished by sending HTTP requests to RESTBase.

Deploying

To scb

To Kubernetes

Depending on what needs to be changed in a Kubernetes deploy of changeprop, edits might need to take place in one of two locations - the Helmfile or the Helm chart. Whether the change is to the Helm chart itself, or the Helmfile that configures it, the deploy process to Kubernetes is the same.

Applying changes

Once your changes have been reviewed and merged by giving a +2 to your change when no rebase is required:

  • a user with root will need to ssh to deploy1001 and sudo to root
  • cd to /srv/deployment-charts/helmfile.d/services/staging/changeprop/.
  • Do a git log -n1 to ensure that your change has been merged and is present in the local checkout of the repo.
  • Prepare your environment by doing source .hfenv
  • Check the impact of your changes on configuration files etc by running helmfile diff
  • If everything looks okay, run a helmfile sync and then monitor kubectl get pods to ensure everything comes back up healthy

Helmfile changes

For the purposes of this section we'll assume that all changes will be against the staging environment. Helmfile changes happen in the helmfile.d section of the deployment-charts repository. Typically a change to this section will relate to changing an existing configured value for deployed instances (ie: adding Kafka or Redis servers, changing the Varnish multicast IP address).

Helm chart changes

Testing

changeprop can be tested by issuing events to Kafka that changeprop will consume. An example test command against the resource_change topic for the k8s staging cluster is: cat mw_purge_example.json | kafkacat -b localhost:9092 -p 0 -t 'staging.resource_change'.

All IDs in these examples are random UUIDs. Not varying UUID between tests runs the risk of being seen as a duplicate event and being skipped. The "dt" field should also be changed to be close to the current time and date, as changeprop will not take action on older events.

mw_purge

{"$schema":"/resource_change/1.0.0","meta":{"dt": "2020-04-02T17:16:25Z", "uri":"https://en.wikipedia.org/wiki/Draft:Editta_Braun","id":"22350141-bbe2-488d-9f73-a1aa6094ac5c","domain":"en.wikipedia.org","stream":"resource_change"},"tags":["purge"]}

null_edit

{"$schema":"/resource_change/1.0.0","meta":{"uri":"https://fr.wikipedia.org/wiki/Oribiky","id":"b92d40b0-3206-469d9615-2fbf61a04418","dt":"2020-04-02T17:16:28Z","domain":"fr.wikipedia.org","stream":"resource_change"},"tags":["null_edit"]}

Where it lives

Related pages

  • Changeprop emerged out of the older and now decommissioned Event_Platform/EventBus system. This page is largely out of date and does not represent the current system.