You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Caching overview

From Wikitech-static
(Redirected from LVS and Varnish)
Jump to navigation Jump to search
"CDN" redirects here. For the Toolforge CDN, see Help:Toolforge/Web#External assets.

This page documents the traffic routing and caching infrastructure from a high level. It details which layers of the infrastructure exist in edge caching PoPs, which exist in core data centers, and how traffic flows through them.

Cache software

Wikipedia request flow

As of April 2022, our stack is comprised of:

The frontend layer is effectively equally distributed and is responsible for traffic capacity. Each server at this layer is effectively identical and is statistically very likely to hold a copy of the same HTTP responses in-memory.

The backend layer is distributed by request hash (e.g. the URL and other metadata), and it is responsible for content capacity. Each server is assigned a subset of URLs, and thus together is able to hold a diverse and long-tail of HTTP responses.

As a single server cannot handle all end-user traffic for a single peak-popularity page, the frontend layer serves an important role ahead of the backend. The frontend is responsible for absorbing and coalescing concurrent requests for the same URL when it is absent from the cache, so that it only places minimal demand on the (one) backend assigned for that URL.

Refer to #History for earlier iterations of our caching software.

Retention

Web browsers first hit the LVS load balancers.

LVS distributes traffic to the edge frontend cluster powered by HAProxy and Varnish cache. As of June 2022, the frontend cache is capped to 1 day with a 7 day keep for benefit of HTTP-304 via IMS/INM (wikimedia-frontend.vcl).

Misses from the frontend are hashed to the edge backend cluster powered by ATS cache. Since April 2020, the ATS backend TTL is capped to 24 hours (T249627, trafficserver/backend.pp).

Misses from the backend are distributed to the MediaWiki app servers. Since July 2016, the Cache-Control max-age for page views is 14 days (T124954, $wgCdnMaxAge). Since May 2021, the wikitext parser cache retains entries for 21 days (change 685181, wgParserCacheExpireTime, wmf-config).

Invalidating content

For Varnish:

  • When pages are edited, their canonical url is proactively purged by MediaWiki (via Kafka and Purged) .

For ParserCache:

  • Values in ParserCache are verifiable by revision ID. Edits will naturally invalidate it.
  • The TTL is enforced through a daily maintenance script, schedule via Puppet class misc::maintenance::parsercachepurging.

Routing

See also: Global traffic routing
This diagram is for the "text" cache cluster but the traffic for "upload" operates similarly as well.

When LVS balances traffic to port 443 (HAProxy), it uses a hash of the client IP to help with TCP Fast Open and SSL session persistence.

Within the caching layer (cp servers), the jump from HAProxy:443 to Varnish:80 is direct on the local host.

However, the jump from Varnish (frontend, port 80) to Apache Traffic Server (backend, port 3128) is different: For that jump, we hash on the URL (and other req meta-data) when balancing to the backends to divide the cache space among all machines, and thus the request typically moves from one machine to another within the same cluster.

Cache clusters

Current cache clusters in all data centers:

  • cache_text - Primary cluster for MediaWiki and various app/service (e.g. RESTBase, phabricator) traffic
  • cache_upload - Serves upload.wikimedia.org and maps.wikimedia.org exclusively (images, thumbnails, map tiles)

Former clusters (no longer exist):

  • cache_bits - Used to exist just for static content and ResourceLoader, now decommed (traffic went to cache_text)
  • cache_mobile - Was like cache_text but just for (m|zero)\. mobile hostnames, now decommed (traffic went to cache_text)
  • cache_parsoid - Legacy entrypoint for parsoid and related *oid services, now decommend (traffic goes via cache_text to RestBase)
  • cache_maps - Served maps.wikimedia.org exclusively, which is now serviced by cache_upload
  • cache_misc - Miscellaneous lower-traffic / support services (e.g. phabricator, metrics, etherpad, graphite, etc). Now moved to cache_text.

Headers

See also MediaWiki HTTP cache headers

X-Cache

X-Cache is a comma-separated list of cache hostnames with information such as hit/miss status for each entry. The header is read right-to-left: the rightmost is the outermost cache, things to the left are progressively deeper towards the applayer. The rightmost cache is the in-memory cache, all others are disk caches.

In case of cache hit, the number of times the object has been returned is also specified. Once "hit" is encountered while reading right to left, everything to the left of "hit" is part of the cached object that got hit. It's whether the entries to the left missed, passed, or hit when that object was first pulled into the hitting cache. For example: 

X-Cache: cp1066 hit/6, cp3043 hit/1, cp3040 hit/26603

An explanation of the possible information contained in X-Cache follows.

Might not talk to other servers

  • hit: a cache hit in cache storage. There was no need to query a deeper cache server (or the applayer, if already at the last cache server). Hits could need reaching an inner layer if content is stale and must-revalidate is set. In this scenario the cache server sends a conditional request to an inner layer and if a 304 Not Modified is obtained the response is sent from the cache.
  • int: locally-generated response from the cache. For example, a 301 redirect. The cache did not use a cache object and it didn't need to contact another server. Backend errors will trigger an int response as well. let's consider a backend responding with a 429 without a response body, the cache will internally generate an error response after contacting the applayer.

Talking to other servers

  • miss: the object might be cacheable, but we don't have it
  • pass: the object was uncacheable, talk to a deeper level

Some subtleties on pass: different caches (eg: in-memory vs. on-disk) might disagree on whether the object is cacheable or not. A pass on the in-memory cache (for example, because the object is too big) could be a hit for an on-disk cache. Also, it's sometimes not clear that an object is uncacheable till the moment we fetch it. In that case, we cache for a short while the fact that the object is uncachable. In Varnish terminology, this is a hit-for-pass.

If we don't know an object is uncacheable until after we fetch it, it's initially identical to a normal miss. Which means coalescing, other requests for the same object will wait for the first response. But after that first fetch we get an uncacheable object, which can't answer the other requests which might have queued. Because of that they all get serialized and we've destroy the performance of hot (high-parallelism) objects that are uncacheable. hit-for-pass is the answer to that problem. When we make that first request (no knowledge), and get an uncacheable response, we create a special cache entry that says something like "this object cannot be cached, remember it for 10 minutes" and then all remaining queries for the next 10 minutes proceed in parallel without coalescing, because it's already known the object isn't cacheable.

The content of the X-Cache header is recorded for every request in the webrequest log table.

Functionalities provided by cache backends

The following functionalities are provided by all cache backends:

  • Path normalization
  • Pass everything which is not GET or HEAD
  • Pass X-Wikimedia-Debug
  • Pass Authorization
  • Pass Set-Cookie responses
  • Pass CC:private, no-cache, no-store
  • Pass X-MISS2PASS
  • Performance hack to assign a single Vary slot for HFP to logged in users
  • Provides custom error html if error response has no body
  • Set X-Cache-Int
  • Compress compressible things if the origin didn't already
  • Set various TTL caps
  • Return 403 to client IPs not in wikimedia_trust
  • Unset Accept-Encoding to avoid some corner-cases (see T125938)
  • Unset Public-Key-Pins Public-Key-Pins-Report-Only

Specific to cache_text:

  • Pass the beta variant of the mobile site
  • Pass cxserver.wikimedia.org
  • Request mangling for MediaWiki (keywords: Host, X-Dt-Host, X-Subdomain)
  • Request mangling for RESTBase (/api/rest_v1/ -> /v1/)
  • Request mangling for w.wiki (send to meta.wikimedia.org/wiki/Special:UrlRedirector)
  • Vary slotting for PHP7 (X-Seven)
  • Vary slotting for X-Forwarded-Proto on 301/302
  • Reduce TTL to 60s for mobileaction= / useformat=

Specific to cache_upload:

  • Storage binning to try workaround scalability issues of -sfile
  • Request mangling for X-MediaWiki-Original
  • Disable streaming if Content-Length is missing
  • Pass small objects
  • Pass objects >= ~1GB
  • Pass 200 responses with CL:0 (T144257)

Specific to cache_misc:

  • Pass objects >= ~1GB
  • Disable streaming if Content-Length is missing
  • Cache requests with google analytics cookies and our own global WMF-Last-Access, WMF-Last-Access-Global GeoIP, and CP cookies

History

An overview of notable events and changes to our caching infrastructure.

2022

In April 2022, we replaced ATS with HAProxy for TLS termination and HTTP2 (T290005). This changed the stack to: HAProxy for TLS termination, Varnish frontend, and ATS backend.

2020

In June 2022, the Purged service was introduced. MediaWiki no longer uses multicast HTCP purging, but instead produces Kafka events for purging URLs, which local Purged instances on Varnish and ATS servers consume and apply by producing local PURGE requests.

In April 2020, a year after switching from Varnish to ATS as cache backend, the TTL was re-enabled and lowered from the 7 days set in 2016, down to 24 hours (T249627). With Varnish frontend also at 1 day and a grace-keep of 7 days, this means frontend objects may outlive backend ones.

2019

In 2019, we adopted the "ATS sandwich" featuring Apache Traffic Server (ATS) as both TLS terminator and as backend cache, thus discontinuing Nginx- ("nginx minus") and Varnish backend. This changed the stack to:

  • ATS for TLS termination (ats-tls),
  • Varnish frontend (varnish-fe), and
  • ATS backend (ats-be).

It was explored to evolve the ATS-TLS layer to subsume the responsibilities of Varnish-frontend one day.

Prior to 2019, the stack for many years involved Nginx- for TLS termination and HTTP2, Varnish as frontend, and a second Varnish layer as cache backend. As such, in older documentation "Varnish" might sometimes also refer to the cache backend.

2016

Prior to 2019, we used Nginx- for TLS termination, Varnish as cache frontend, and a second Varnish layer for the backend cache. In older documentation "Varnish" thus might thus also be referring to the cache backend as well.

In 2016, we decreased the max object ttl in Varnish from the long-standing 31 days down to 1 day for Varnish frontends, and 14 days for Varnish backends and MediaWiki (T124954). The parser cache remains at 31 days.

In 2016, we deployed HTTP/2 support to the Wikimedia CDN, which was at the time comprised of Nginx- and Varnish (T96848).

2013

Prevent white-washing of expired page-view HTML. Various static aspects of a page are not tracked or versions, as such, when the max-age expires, a If-Not-Modified must not return true after expiry even if the database entry of the wiki page was unchanged (T46570).

Further reading

Retrieved from "https://wikitech-static.wikimedia.org/w/index.php?title=Caching_overview&oldid=599516#LVS_and_Varnish_infrastructure"