You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Redis
Redis is used in Wikimedia production for:
- changeprop (role::redis::misc)
- stashing user sessions ($wgSessionCacheType) and the data stored in the main stash ($wgMainStash).
- Maps
- As a cache and queue backend in ORES
- Receiver of sampled profile data from PHP, as part as the sampling/profiling pipeline (Arc Lamp).
- stores lists of articles for new editors to edit as part of the GettingStarted MediaWiki extension
MediaWiki-Vagrant and MediaWiki-Vagrant in Cloud VPS are configured by default to use redis for $wgMainCacheType, $wgSessionCacheType, etc.
Role redis::misc
The role redis::misc is for our general purpose master-replica cluster in eqiad and codfw DCs. Each rdb* node has 5 instances (ports 6378, 6379, 6380, 6381, 6382) because redis is single threaded. A mapping of usages is below
The servers are setup as 2 independent pairs. This is for HA purposes and it's up to the application to use it that way. Conversely not all applications are able to do so. changeprop/changeprop-jobqueue are able to do so, but e.g. ORES(cache and queue) or docker-registry use just 1 of the 2 pairs.
Pair 1
| Port | redis db | Usage |
|---|---|---|
| 6378 | 0 | ORES cache |
| 6379 | 0 | changeprop/cpjobqueue/api-gateway |
| 6380 | 0 | ORES queue |
| 6381 | 0 | unallocated |
| 6382 | 0 | Reserved for docker-registry |
Pair 2
| Port | redis db | Usage |
|---|---|---|
| 6378 | 0 | Reserved for ORES cache |
| 6379 | 0 | changeprop/cpjobqueue/api-gateway |
| 6380 | 0 | Reserved for ORES queue |
| 6381 | 0 | unallocated |
| 6382 | 0 | docker-registry |
Servers
Each master has its respected slave. Masters use odd numbers (e.g. rdb1005) and slaves an even one (e.g. rdb1006). Master-slave instances use the same ports e.g. rdb1005:6379 is the master of rdb1006:6379
eqiad:
- rdb1005 (m) - rdb1006 (s)
- rdb1009 (m) - rdb1010 (s)
codfw:
- rdb2003 (m) - rdb2004 (s)
- rdb2005 (m) - rdb2006 (s)
Services
Change propagation (or changeprop) is a service that runs on Kubernetes nodes by listening to topics on Kafka for events, and then translating them into http requests to various systems. It is also responsible for cache evictions to happen on all services like RESTBase. Changeprop talks to redis via Nutcracker.
Related puppet code
hieradata/role/common/redis/misc/master.yamlhieradata/role/common/redis/misc/slave.yamlmodules/role/manifests/redis/misc/master.ppmodules/role/manifests/redis/misc/slave.pp
Other Info
- Instance passwords can be easily found under
/etc/redis/<instance>.conf - Grafana redis::misc Dashboard
Using Redis
Connecting
redis-cli is installed on all servers where redis-server is installed. This will leave you at a redis prompt where you can enter commands interactively.
Some useful commands
AUTH <somepass>authenticateINFOstatus information, including:
# Replication role:slave master_host:10.64.0.24 master_port:6379 master_link_status:up master_last_io_seconds_ago:0 <snip> # Keyspace db0:keys=9351936,expires=9291239,avg_ttl=0
KEYS <pattern-here>list of all keys matching the given pattern. Use this sparingly! This query could take seconds to completeQUITcloses the connection.
Using Redis from other Services
Some services may require or be able to use Redis, and this Redis cluster is appropriate for that.
As noted above, each pair of Redis servers in each data center have five separate instances on different ports, a majority of which are not in use; the first step to using the Redis server in production service is to choose an unused instance/port pair which can be located by examining Hiera data for what is currently in use: a relatively straight forward way to do this is to use git grep '\Wrdb[12]' within a Puppet tree, which shows every use of an rdb address. A similar procedure may be used to find a port that is unallocated.
Once a port/host combination for each datacenter is chosen, it is as simple as referring to those from the Puppet state which will use them.
Using Redis from a service requires a password; the password may be obtained from the Hiera key ::passwords::redis::main_password in hieradata/role/common/redis/misc/master.yaml in the private repository. It is currently the convention to introduce a new private Hiera key to store the password for your service's use, however this is obviously inefficient and subject to change.
Other references
Commands are easy, they all depend on the data type (hash, set, list, etc). Here's a quick reference.
Configuration is likewise pretty straightforward with perhaps the exception of the snapshotting, aof and memory settings; here's the sample config file.
See also
- memcached
- nutcracker (AKA twemproxy), the proxy used by all application servers to contact memcached (but not redis as of 2015, except it does again as of 2016)