You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Memcached: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Krinkle
(+{{Navigation Wikimedia infrastructure|expand=mw}})
imported>Elukey
Line 1: Line 1:
{{Navigation Wikimedia infrastructure|expand=mw}}
{{warning|1= This page is under construction, please be patient :)}}
The Memcache setup changed in 2015 to be multi-datacenter aware. See also:
* [[mw:Manual:WANObjectCache.php]]
* [[nutcracker]] (memcached proxy on all application servers)


== Disabling broken servers ==
[https://github.com/memcached/memcached Memcached] is deployed in several clusters on our infrastructure:


When a memcache server is offline, it should be entirely removed from the configuration (if permanent) or commented out (if temporary.)  Unlike the old slot format config file, the new format doesn't require you to keep an identical number of members deployed in the memcached pool.  Please note that one shouldn't add and remove a lot of members from this list in a short period of time, as the consistent hashing may result in a server being unpooled, then repooled to cause the keys to remap directly back to the same server, resulting in undesired performance.  So if a server is down, fix it ASAP.  If it cannot be rebooted and fixed ASAP, then it should be removed.  If it is removed, it may be prudent to leave it unpooled for 24 hours? (Aaron advises we have some items that can cache in mc for 24 hours.)
* MediaWiki Object Cache (mc10XX, mc20XX)
* Thumbor
* WMCS
* Wikitech on Cloud/Labs
* Swift


The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.
This page contains information about the MediaWiki Object cache, running on mc10XX hosts (in eqiad) and mc20XX hosts (in codfw).
<tt>mediawiki-config/wmf-config/mc-eqiad.php</tt>
<tt>mediawiki-config/wmf-config/mc-pmtpa.php</tt>


Memcached installation is handed via the puppet memcached role; and are currently deployed to dedicated memcache systems (mcX and mcXXXX).
== MediaWiki libraries/support ==
 
{{Navigation Wikimedia infrastructure|expand=mw}}
 
The Memcache setup changed in 2015 to be multi-datacenter aware. See also:
You can test the servers to see which memcached are functioning and which are not with the following on fenari:
* [[mw:Manual:WANObjectCache.php]]
<tt>cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php</tt>
*<s>[[nutcracker]] (memcached proxy on all application servers)</s> (deprecated as of March 2019)
 
*[[Mcrouter]] (memcached proxy on all application servers)
Note that all servers that are up will return <tt>incr: 100  get: 100</tt> and servers that are down will return <tt>incr: 0  get: 0</tt>.
 
== Editing mc.php ==
 
The memcached configuration php files are maintained in gerrit, under the operations/mediawiki-config branch.
<tt>mediawiki-config/wmf-config/mc-eqiad.php</tt>
<tt>mediawiki-config/wmf-config/mc-pmtpa.php</tt>
 
If you want to add a server from the spare list to the active list, please test it first. You can run on fenari
 
  <tt>cd /home/w/common/wmf-deployment/maintenance; mwscript mctest.php enwiki ip-address-here:11000</tt>


(it runs on a nonstandard port). As above, servers that are up will return incr: 100 get: 100 and servers that are down will return incr: 0 get: 0.
=== WANObjectCache ===
TODO


Once you finish updating the file, you must '''git commit''' and then '''git review'''.  You should be logged in as your own username, as your user keys will be checked.  Then you need to git fetch and merge on fenari in the /home/wikimedia/common/wmf-deployment directory and sync out changes.
=== Mcrouter ===
Each MediaWiki api/appserver sees memcached through a local proxy called mcrouter. This daemon is configured with all the mc[1,2]0XX hosts as shards, and applies consistent hashing on the key name to know where to store it. The following example should help clarifying how things flow on this layer.


== Deleting a memcache key ==
The MediaWiki translate extension stores the key WANCache:v:metawiki:translate-groups in memcached, using the WANObjectCache as library. For example, a GET command to retrieve the key's value is sent from MediaWiki to localhost:11213, where mcrouter is listening, and the command gets routed to mc1022. MediaWiki it totally ignorant about the mc[1,2]0XX host, it only knows about sending commands to a localhost port. A mcrouter admin command helps figure out where the key gets routed:<syntaxhighlight lang="bash">
elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2
VALUE __mcrouter__.route(get,WANCache:v:metawiki:translate-groups) 0 16
10.64.0.83:11211
END


<pre>
elukey@mw1345:~$ dig -x 10.64.0.83 +short
mwscript mcc.php --wiki=$wiki
mc1022.eqiad.wmnet.
> delete $wiki:$key
</syntaxhighlight>Some things to notice:
> exit
</pre>


If the deletion is successful, no output will be shown.  
* The special prefix __mcrouter__.route is interpreted by mcrouter as admin command, that in this case returns the proxy target of the consistent hashing of the key name.
* mcrouter listen on port 11213 on every MediaWiki hosts, meanwhile on every mc10XX host memcached listens on 11211.


== See also ==
To get a key and dump it to a file it is sufficient to:<syntaxhighlight lang="bash">
* [[Redis]]: "Redis is used in production for the MediaWiki job queue and user sessions"
elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt
elukey@mw1345:~$ du -hs dump.txt
380K dump.txt
</syntaxhighlight>In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content.


== How Memcached works ==
TODO
[[Category:How-To]]
[[Category:How-To]]
[[Category:Caching]]
[[Category:Caching]]
[[Category:MediaWiki production]]
[[Category:MediaWiki production]]

Revision as of 16:47, 13 March 2019

Memcached is deployed in several clusters on our infrastructure:

  • MediaWiki Object Cache (mc10XX, mc20XX)
  • Thumbor
  • WMCS
  • Wikitech on Cloud/Labs
  • Swift

This page contains information about the MediaWiki Object cache, running on mc10XX hosts (in eqiad) and mc20XX hosts (in codfw).

MediaWiki libraries/support

The Memcache setup changed in 2015 to be multi-datacenter aware. See also:

WANObjectCache

TODO

Mcrouter

Each MediaWiki api/appserver sees memcached through a local proxy called mcrouter. This daemon is configured with all the mc[1,2]0XX hosts as shards, and applies consistent hashing on the key name to know where to store it. The following example should help clarifying how things flow on this layer.

The MediaWiki translate extension stores the key WANCache:v:metawiki:translate-groups in memcached, using the WANObjectCache as library. For example, a GET command to retrieve the key's value is sent from MediaWiki to localhost:11213, where mcrouter is listening, and the command gets routed to mc1022. MediaWiki it totally ignorant about the mc[1,2]0XX host, it only knows about sending commands to a localhost port. A mcrouter admin command helps figure out where the key gets routed:

elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2
VALUE __mcrouter__.route(get,WANCache:v:metawiki:translate-groups) 0 16
10.64.0.83:11211
END

elukey@mw1345:~$ dig -x 10.64.0.83 +short
mc1022.eqiad.wmnet.

Some things to notice:

  • The special prefix __mcrouter__.route is interpreted by mcrouter as admin command, that in this case returns the proxy target of the consistent hashing of the key name.
  • mcrouter listen on port 11213 on every MediaWiki hosts, meanwhile on every mc10XX host memcached listens on 11211.

To get a key and dump it to a file it is sufficient to:

elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt
elukey@mw1345:~$ du -hs dump.txt
380K	dump.txt

In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content.

How Memcached works

TODO