You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Memcached: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Krinkle
imported>Krinkle
(Krinkle moved page Memcached to Memcached for MediaWiki)
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{warning|1= This page is under construction, please be patient :)}}
#REDIRECT [[Memcached for MediaWiki]]
{{Navigation Wikimedia infrastructure|expand=mw}}
 
[https://github.com/memcached/memcached Memcached] is deployed in several clusters on our infrastructure:
 
* MediaWiki Object Cache (mc10XX, mc20XX)
* Thumbor
* WMCS
* Wikitech on Cloud/Labs
* Swift
 
This page contains information about the MediaWiki Object cache, running on mc10XX hosts (in eqiad) and mc20XX hosts (in codfw).
 
See also:
*[[Mcrouter]] (memcached proxy on all application servers)
 
== WANObjectCache ==
'''WANObjectCache''' is the higher-level abstraction layer in MediaWiki PHP that deals with multi-datacenter concerns and Mcrouter. It builds on top of '''BagOStuff''' which is generic key-value class used to abstract the Memcached protocol itself.
 
See also:
* [https://doc.wikimedia.org/mediawiki-core/master/php/classWANObjectCache.html#details WANObjectCache high-level documentation]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/includes/libs/objectcache/README.md WANObjectCache metrics and internal details]
* [[mw:Manual:Caching]], documentation of what kind of data is stored, what the keys relate to etc.
 
=== High level ===
* '''Like a replica.''' Historically, it was common for MediaWiki to populate its cache during HTTP write actions. This meant that in a single DC setup it could loosely be expected that the cache was as up-to-date as the master DB. As part of the multi-dc effort, this was changed starting in 2015. There is now generally no proactive setting of values during HTTP write actions. Instead, values are generally computed based on information from replica DBs, and computed on-demand using the <code>getWithSet(key, ttl, callable)</code> idiom. This means the application generally only expects cache values to be as up to date as a replica DB would be.
* '''No synchronisation.''' MediaWiki's WANObjectCache layer does not require synchronisation of cached values cross data centers. Instead, it considers each datacenter's Memcached cluster as independent. Each populating its own values as-needed on dc-local app servers from dc-local replica DBs.
* '''Tombstones (broadcasted purge)''' During HTTP write actions, MediaWiki asks WANObjectCache to purge cache keys of which it has modified the source data. These purges take the form of a short-lived Memcached key known as a "tombstone". We do not use the DELETE command because we want each data center to be able to populate its memcached independent, thus requiring no cross-dc master database connection, thus reading from a local replica, thus values ingested in the cache may be as stale as a replica can be. Implementing a Memcached purge as DELETE would mean both in the same DC and other DCs, the same key could be re-populated immediately with the same stale value we just deleted. Instead, WANObjectCache formulates its purge as a SET operation that stores a placeholder value known as a "tombstone" (lasts for approx. 10 seconds).
* '''Interim values''' Upon seeing such tombstone, WANObjectCache acts much like a cache miss, except that the newly computed value is not written back over the tombstone (as it may be stale). Instead, to avoid a recompute stampede these maybe-stale values are stored as an "interim value" in a sister key.
 
=== Memcached commands ===
 
Intra-dc:
 
* Read traffic from the <code>getWithSet</code> idiom results in a <code>GETS</code> command (getMulti) that fetches the main key, plus any sister keys that might exist.
* Write trafffic from the <code>getWithSet</code> idiom results in either <code>ADD</code> if the key was known to be absent, or <code>Memcached->mergeViaCas</code> if a value existed but either required (or was elected for) regeneration.
 
Cross-dc:
 
* Purge traffic uses the <code>/*/mw-wan/</code> prefix to tell Mcouter to broadcast this to other pools and clusters as well. The actual command is generally <code>SET</code> as it needs to induce a "hold-off" period using the tombstone (per the above). In rare cases where a hold-off is not needed (e.g. if the purge is not related to a DB write), then the broadcasted event will use <code>DELETE</code>
 
=== Magic numbers ===
: ''Last updated: May 2020.''
 
* Tombstone (aka ''hold-off TTL''): '''11 seconds'''
* Interim value: '''1 second'''
 
== Mcrouter ==
Each MediaWiki api/appserver sees memcached through a local proxy called [[Mcrouter]]<ref>[https://engineering.fb.com/web/introducing-mcrouter-a-memcached-protocol-router-for-scaling-memcached-deployments/ Introducing mcrouter: A memcached protocol router for scaling memcached deployments]</ref>. This daemon is configured with all the mc[1,2]0XX hosts as shards, and applies consistent hashing on the key name to know where to store it. The following example should help clarifying how things flow on this layer.
 
The MediaWiki translate extension stores the key WANCache:v:metawiki:translate-groups in memcached, using the WANObjectCache as library. For example, a GET command to retrieve the key's value is sent from MediaWiki to localhost:11213, where mcrouter is listening, and the command gets routed to mc1022. MediaWiki it totally ignorant about the mc[1,2]0XX host, it only knows about sending commands to a localhost port. A mcrouter admin command helps figure out where the key gets routed:<syntaxhighlight lang="bash">
elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2
VALUE __mcrouter__.route(get,WANCache:v:metawiki:translate-groups) 0 16
10.64.0.83:11211
END
 
elukey@mw1345:~$ dig -x 10.64.0.83 +short
mc1022.eqiad.wmnet.
</syntaxhighlight>Some things to notice:
 
* The special prefix __mcrouter__.route is interpreted by mcrouter as admin command, that in this case returns the proxy target of the consistent hashing of the key name.
* mcrouter listen on port 11213 on every MediaWiki hosts, meanwhile on every mc10XX host memcached listens on 11211.
 
To get a key and dump it to a file it is sufficient to:<syntaxhighlight lang="bash">
elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt
elukey@mw1345:~$ du -hs dump.txt
380K dump.txt
</syntaxhighlight>In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content.
 
== How Memcached works ==
TODO
[[Category:How-To]]
[[Category:Caching]]
[[Category:MediaWiki production]]

Latest revision as of 23:46, 27 July 2020