Cache-Control page views is 14days ([[Mw:Manual:$|$]];[://mediawiki-config//][[Mw:Manual:$wgParserCacheExpireTime|wgParserCacheExpireTime]][https://github.com/wikimedia/operations-mediawiki-config/blob/bf4ee52d1c1b9a6777168969908d6553bf820508/wmf-config/InitialiseSettings.php#L10714-L10716 wmf-config])
=== Invalidating content ===
=== Invalidating content ===
* When pages are edited, their canonical url is proactively purged in Varnish by MediaWiki.
* When pages are edited, their canonical url is proactively purged by MediaWiki .
Values in ParserCache are verifiable by revision ID. Edits will naturally update it.
* puppet: manifests/misc/maintenance.pp
Values in ParserCache are verifiable by revision ID. Edits will naturally it.
** class misc::maintenance::parsercachepurging
*** Set to 22 days (<code>expire age=2592000</code>)
== Past events ==
, the , the of Varnish [[phab:|]].
* 2013: Prevent white-washing of expired page-view HTML.
** Various static aspects of a page are not tracked or versions, as such, when the max-age expires, a If-Not-Modified must not return true after expiry even if the database entry of the wiki page was unchanged.
** More info: [[phab:T46570|https://phabricator.wikimedia.org/T46570]]
* 2016: Decrease max object ttl in Varnish
** More info: [[phab:T124954|https://phabricator.wikimedia.org/T124954]]
** Varnish frontends changed from 31 days to 1 day.
** Varnish backends changed from 31 days to 14 days.
** MediaWiki max-age changed from 31 days to 14 days.
In a nutshell, the cache hosts run: haproxy for TLS → Varnish cache frontend → ATS cache backend.
This page documents the traffic routing and caching infrastructure from a high level. It details which layers of the infrastructure exist in edge caching PoPs, which exist in core data centers, and how traffic flows through them.
From July 2019 to March 2022 we ran an "ATS sandwhich", with ATS as TLS terminator, Varnish frontend, and a second ATS layer as cache backend.
Prior to 2019, we used Nginx- for TLS termination, Varnish frontend, and used a second Varnish layer for the backend cache. In older documented "Varnish" thus might also be referring to the cache backend.
Web browsers first hit the LVS load balancers.
LVS distributes traffic to the edgefrontend cluster powered by HAProxy and Varnish cache. As of June 2022, the frontend cache is capped to 1 day with a 7 day keep for benefit of HTTP-304 via IMS/INM (wikimedia-frontend.vcl).
Misses from the frontend are hashed to the edgebackend cluster powered by ATS cache. Since April 2020, the ATS backend TTL is capped to 24 hours (T249627, trafficserver/backend.pp).
When pages are edited, their canonical url is proactively purged by MediaWiki (via Kafka and Purged) .
Values in ParserCache are verifiable by revision ID. Edits will naturally invalidate it.
The TTL is enforced through a daily maintenance script, schedule via Puppet class misc::maintenance::parsercachepurging.
This information is outdated. We no longer run Nginx for frontend TLS. We also no longer expose port 80 directly on cache frontends.
When LVS balances traffic to ports :80 (varnish), and :443 (nginx), it uses a hash of the client IP to help with TCP Fast Open and SSL session persistence respectively.
Within the caching layer (cp#xxx machines), the jump from nginx:443 to varnish:80 is direct on the local host.
However, the jump from varnish:80 (frontend) to varnish:3128 (backend) is different: for that jump, we hash on the URL (and other req meta-data) when balancing to the backends to divide the cache space among all machines, and thus the request typically moves from one machine to another within the same cluster.
X-Cache is a comma-separated list of cache hostnames with information such as hit/miss status for each entry. The header is read right-to-left: the rightmost is the outermost cache, things to the left are progressively deeper towards the applayer. The rightmost cache is the in-memory cache, all others are disk caches.
In case of cache hit, the number of times the object has been returned is also specified. Once "hit" is encountered while reading right to left, everything to the left of "hit" is part of the cached object that got hit. It's whether
the entries to the left missed, passed, or hit when that object was first pulled into the hitting cache. For example:
An explanation of the possible information contained in X-Cache follows.
Not talking to other servers
hit: a cache hit in cache storage. There was no need to query a deeper cache server (or the applayer, if already at the last cache server)
int: locally-generated response from the cache. For example, a 301 redirect. The cache did not use a cache object and it didn't need to contact another server
Talking to other servers
miss: the object might be cacheable, but we don't have it
pass: the object was uncacheable, talk to a deeper level
Some subtleties on pass: different caches (eg: in-memory vs. on-disk) might disagree on whether the object is cacheable or not. A pass on the in-memory cache (for example, because the object is too big) could be a hit for an on-disk cache. Also, it's sometimes not clear that an object is uncacheable till the moment we fetch it. In that case, we cache for a short while the fact that the object is uncachable. In Varnish terminology, this is a hit-for-pass.
If we don't know an object is uncacheable until after we fetch it, it's initially identical to a normal miss. Which means coalescing, other requests for the same object will wait for the first response. But after that first fetch we get an uncacheable object, which can't answer the other requests which might have queued. Because of that they all get serialized and we've destroy the performance of hot (high-parallelism) objects that are uncacheable. hit-for-pass is the answer to that problem. When we make that first request (no knowledge), and get an uncacheable response, we create a special cache entry that says something like "this object cannot be cached, remember it for 10 minutes" and then all remaining queries for the next 10 minutes proceed in parallel without coalescing, because it's already known the object isn't cacheable.
The following functionalities are provided by all cache backends:
Pass everthing which is not GET or HEAD
Pass X-Wikimedia-Debug and X-Wikimedia-Security-Audit
Pass Set-Cookie responses
Pass CC:private, no-cache, no-store
Performance hack to assign a single Vary slot for HFP to logged in users
Provides custom error html if error response has no body
Compress compressible things if the origin didn't already
Set various TTL caps
Return 403 to client IPs not in wikimedia_trust
Unset Accept-Encoding to avoid some corner-cases (see T125938)
Unset Public-Key-Pins Public-Key-Pins-Report-Only
Specific to cache_text:
Pass the beta variant of the mobile site
Request mangling for MediaWiki (keywords: Host, X-Dt-Host, X-Subdomain)
Request mangling for RESTBase (/api/rest_v1/ -> /v1/)
Request mangling for w.wiki (send to meta.wikimedia.org/wiki/Special:UrlRedirector)
Vary slotting for PHP7 (X-Seven)
Vary slotting for X-Forwarded-Proto on 301/302
Reduce TTL to 60s for mobileaction= / useformat=
Specific to cache_upload:
Storage binning to try workaround scalability issues of -sfile
Request mangling for X-MediaWiki-Original
Disable streaming if Content-Length is missing
Pass small objects
Pass objects >= ~1GB
Pass 200 responses with CL:0 (T144257)
Specific to cache_misc:
Pass objects >= ~1GB
Disable streaming if Content-Length is missing
Cache requests with google analytics cookies and our own global WMF-Last-Access, WMF-Last-Access-Global GeoIP, and CP cookies
An overview of notable events and changes to our caching infrastructure.
In April 2022, we replaced ATS with HAProxy for TLS termination and HTTP2 (T290005). This changed the stack to: HAProxy for TLS termination, Varnish frontend, and ATS backend.
In June 2022, the Purged service was introduced. MediaWiki no longer uses multicast HTCP purging, but instead produces Kafka events for purging URLs, which local Purged instances on Varnish and ATS servers consume and apply by producing local PURGE requests.
In April 2020, a year after switching from Varnish to ATS as cache backend, the TTL was re-enabled and lowered from the 7 days set in 2016, down to 24 hours (T249627). With Varnish frontend also at 1 day and a grace-keep of 7 days, this means frontend objects may outlive backend ones.
In 2019, we adopted the "ATS sandwhich" featuring Apache Traffic Server (ATS) as both TLS terminator and as backend cache, thus discontinuing Nginx- ("nginx minus") and Varnish backend. This changed the stack to: ATS for TLS termination (ats-tls), Varnish frontend (varnish-fe), and ATS backend (ats-be). It was explored to evolve the ATS-TLS layer to subsume the responsibilities of Varnish-frontend one day.
Prior to 2019, the stack for many years involved Nginx- for TLS termination and HTTP2, Varnish as frontend, and a second Varnish layer as cache backend. As such, in older documention "Varnish" might sometimes also refer to the cache backend.
In 2016, we decreased the max object ttl in Varnish from the long-standing 31 days down to 1 day for Varnish frontends, and 14 days for Varnish backends and MediaWiki (T124954). The parser cache remains at 31 days.
In 2016, we deployed HTTP/2 support to the Wikimedia CDN, sing the time comprised of Nginx- and Varnish (T96848).
Prevent white-washing of expired page-view HTML. Various static aspects of a page are not tracked or versions, as such, when the max-age expires, a If-Not-Modified must not return true after expiry even if the database entry of the wiki page was unchanged (T46570).