You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Memcached for MediaWiki
This page is about Memcached for MediaWiki.
WANObjectCache is the abstraction layer in MediaWiki PHP that deals with multi-datacenter concerns and Mcrouter. It builds on top of BagOStuff which is the generic key-value class that abstracts the Memcached protocol itself.
- WANObjectCache high-level documentation
- WANObjectCache metrics and internal details
- mw:Manual:Caching: what kind of data is stored here, how it gets there etc.
- Like a replica. There is generally no proactive setting of values during HTTP write actions. Instead, values are computed based on information from replica DBs, and computed on-demand using the
getWithSet(key, ttl, callable)idiom. This means the application generally only expects cache values to be as up to date as a replica DB would be. Historically, it was common for MediaWiki to populate its cache during HTTP write actions instead. This meant that in a single-DC setup it could loosely be expected that the cache was as up-to-date as the master DB. As part of the multi-dc effort, this was changed starting in 2015, and thus its expectations were loosened to that of a replica DB.
- No synchronisation. MediaWiki's WANObjectCache layer does not require synchronisation of cached values across data centers. Instead, it considers each datacenter's Memcached cluster as independent. Each populating its own values as-needed on dc-local app servers from dc-local replica DBs.
- Tombstones (broadcasted purge). During HTTP write actions, MediaWiki asks WANObjectCache to purge cache keys of which it has modified the source data. These purges take the form of short-lived Memcached keys known as "tombstones". We do not use the
DELETEcommand because we want each data center to be able to populate its memcached independently, thus requiring no cross-dc master database connection, thus reading from a local replica, thus values ingested in the cache may be as stale as a replica can be. Implementing a Memcached purge as
DELETEwould mean both in the same DC and other DCs, the same key could be re-populated immediately with the same stale value we just deleted. Instead, WANObjectCache formulates its purge as a
SEToperation that stores a placeholder value known as a "tombstone" (lasts for approx. 10 seconds for local and remote replica DBs to catch up).
- Interim values. Upon seeing such tombstone, WANObjectCache acts much like a cache miss, except that the newly computed value is not written back over the tombstone (as the computed value may be stale). Instead, to avoid a recompute stampede these maybe-stale values are stored as an "interim value" in a sister key which is only kept for a few seconds.
- Read traffic from the
getWithSetidiom results in a
GETScommand (getMulti) that fetches the main key, plus any sister keys that might exist.
- Write trafffic from the
getWithSetidiom results in either
ADDif the key was known to be absent, or
Memcached->mergeViaCasif a value existed but either required (or was elected for) regeneration.
- Purge traffic uses the
/*/mw-wan/prefix to tell Mcouter to broadcast this to other pools and clusters as well. The actual command is generally
SETas it needs to induce a "hold-off" period using the tombstone (per the above). In rare cases where a hold-off is not needed (e.g. if the purge is not related to a DB write), then the broadcasted event will use
Getting revision/page from WANObjectCache key
If you're trying to track down the specific revision text given an SqlBlobStore key, the somewhat convoluted procedure is documented at mw:Manual:Caching#Revision_text.
Last updated: May 2020.
- Tombstone (aka "hold-off TTL"): 11 seconds.
- Interim value: 1 second.
- Main article: Memcached for MediaWiki/mcrouter
Each MediaWiki api/appserver sees memcached through a local proxy called Mcrouter. This daemon is configured with all the
mc[1,2]0XX hosts as shards, and applies consistent hashing on the key name to know where to store it. The following example should help clarifying how things flow at this layer.
The Memcached key
WANCache:v:metawiki:translate-groups (belongs to the Translate extension) is formatted by the WANObjectCache library. When Translate wants to get the value of this key, WANObjectCache will send a
GET command from MediaWiki to
localhost:11213, where Mcrouter is listening. The command is then further routed to
mc1022 (based on key hashing). MediaWiki it totally ignorant about the
mc[1,2]0XX host, it only knows about sending commands to a localhost port. A Mcrouter admin command helps figure out where keys are hashed/routed to:
elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2 VALUE __mcrouter__.route(get,WANCache:v:metawiki:translate-groups) 0 16 10.64.0.83:11211 END elukey@mw1345:~$ dig -x 10.64.0.83 +short mc1022.eqiad.wmnet.
Some things to notice:
- The special prefix
__mcrouter__.routeis intercepted by Mcrouter. These are admin commands, for which proxy returns directly without contacting the Memcached hosts. This function returns the target of the consistent hashing of the key name.
- Mcrouter listens on port 11213 on all MediaWiki app servers, meanwhile on every
mc10XXhost memcached listens on port 11211.
To get a key and dump it to a file it is sufficient to:
elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt elukey@mw1345:~$ du -hs dump.txt 380K dump.txt
In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content.