You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Caching overview: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Greg G (WMF)
(→‎Ways forward: clarification from Kinkle)
 
imported>Krinkle
No edit summary
Line 1: Line 1:
== Picture ==
{{Navigation Wikimedia infrastructure|expand=caching}}
 
== Graph ==
{{See also|LVS and Varnish}}
<pre>
<pre>
  __________________
  __________________
Line 17: Line 18:
     |
     |
     __________________________________________________________
     __________________________________________________________
   |  Front End Caches (Squid or Varnish)                     |
   |  Edge Frontend (Varnish)                                 |
   |    Short lived caches (~10sec, mostly to prevent DDOS) |
   |    Short-lived cache (~10sec, mostly to prevent DDOS)   |
   |    Live in memory                                       |
   |    Stored in memory                                     |
   |__________________________________________________________|
   |__________________________________________________________|
         |
         |
         |
         |
         |
         |
       _____________________________________
       _______________________________
       | Back End Caches (Squid or Varnish) |
       | Edge Backend (Varnish)       |
       |  Longer lived caches                |
       |  Long-lived cache            |
       |  Live on disk                       |
       |  Stored on disk               |
       |_____________________________________|
       |_______________________________|
             |
             |
             |
             |
             |
             |
           _______________________________
           _______________________________________
           | Mediawiki                    |
           | Apaches (MediaWiki PHP)              |
           |  Apaches                    |
          |                                      |
           |  parsercache                 |
           |  * Cache-Control for page view HTML: |
           |   * Currently set at 1 year  |
          |      max-age is 14 days              |
           |_______________________________|
           |  * wikitext parsercache:            |
           |     Expires at 30 days              |
           |_______________________________________|
</pre>
</pre>


== Caching Systems ==
== Varnish ==
* Squid: text, upload in pmtmp
{{See also|Varnish}}
* Varnish: bits, mobile, and upload in eqaid
* Frontends max ttl is 1 day.
* Backends max ttl is 14 days.


== Caching Settings ==
== MediaWiki ==
* Default squid cache expiration is 30 days.
* Default max-age setting in Cache-Control headers on page views is 14 days. ([[Mw:Manual:$wgSquidMaxage|$wgSquidMaxage]]; [https://github.com/wikimedia/operations-mediawiki-config/blob/bf4ee52d1c1b9a6777168969908d6553bf820508/wmf-config/InitialiseSettings.php#L11985-L11986 wmf-config])
* Default parsercache expiration is 365 days
* Default parsercache expiration is 30 days. ([[Mw:Manual:$wgParserCacheExpireTime|$wgParserCacheExpireTime]]; [https://github.com/wikimedia/operations-mediawiki-config/blob/bf4ee52d1c1b9a6777168969908d6553bf820508/wmf-config/InitialiseSettings.php#L10714-L10716 wmf-config] )


== Current method for invalidating cache content ==
=== Invalidating content ===
For Varnish:
* When pages are edited, their canonical url is proactively purged in Varnish by MediaWiki.
For ParserCache:
For ParserCache:
Values in ParserCache are verifiable by revision ID. Edits will naturally update it. 
* puppet: manifests/misc/maintenance.pp
* puppet: manifests/misc/maintenance.pp
** class misc::maintenance::parsercachepurging
** class misc::maintenance::parsercachepurging
*** Set to 30 days
*** Set to 30 days (<code>expire age=2592000</code>)
 
== Ways forward ==
'''option 1:'''
* [https://bugzilla.wikimedia.org/show_bug.cgi?id=44570#c14 reduce parsercache timeout] to size of deployment window (~28 days)
* Tim may have knowledge why [https://bugzilla.wikimedia.org/show_bug.cgi?id=44570#c12 that shouldn't happen]
 
'''option 2:'''
* change away from version numbers in URLs
** maybe use slots or something else
** what to do with things that need to be versioned, eg: skins?
* related: [https://bugzilla.wikimedia.org/show_bug.cgi?id=44570#c15 not using versioned urls] in code that makes it to main html output


'''option 3:'''
== Past events ==
* status quo
* 2013: Prevent white-washing of expired page-view HTML.
** Various static aspects of a page are not tracked or versions, as such, when the max-age expires, a If-Not-Modified must not return true after expiry even if the database entry of the wiki page was unchanged.
** More info: [[phab:T46570|https://phabricator.wikimedia.org/T46570]]
* 2016: Decrease max object ttl in Varnish
** More info: [[phab:T124954|https://phabricator.wikimedia.org/T124954]]
** Varnish frontends changed from 31 days to 1 day.
** Varnish backends changed from 31 days to 14 days.
** MediaWiki max-age changed from 31 days to 14 days.


'''option 4:'''
[[Category:Caching]]
* ???

Revision as of 20:54, 19 July 2016

Graph

 __________________
| browser/the webz |
|__________________|
    |
    |
    |
  ____________________
 |  LVS               |
 |    (load balancer) |
 |____________________|
     |
     |
     |
     |
    __________________________________________________________
   |  Edge Frontend (Varnish)                                 |
   |     Short-lived cache (~10sec, mostly to prevent DDOS)   |
   |     Stored in memory                                     |
   |__________________________________________________________|
         |
         |
         |
       _______________________________
      | Edge Backend (Varnish)        |
      |  Long-lived cache             |
      |  Stored on disk               |
      |_______________________________|
            |
            |
            |
           _______________________________________
          | Apaches (MediaWiki PHP)               |
          |                                       |
          |   * Cache-Control for page view HTML: |
          |      max-age is 14 days               |
          |   * wikitext parsercache:             |
          |      Expires at 30 days               |
          |_______________________________________|

Varnish

  • Frontends max ttl is 1 day.
  • Backends max ttl is 14 days.

MediaWiki

Invalidating content

For Varnish:

  • When pages are edited, their canonical url is proactively purged in Varnish by MediaWiki.

For ParserCache: Values in ParserCache are verifiable by revision ID. Edits will naturally update it.

  • puppet: manifests/misc/maintenance.pp
    • class misc::maintenance::parsercachepurging
      • Set to 30 days (expire age=2592000)

Past events

  • 2013: Prevent white-washing of expired page-view HTML.
    • Various static aspects of a page are not tracked or versions, as such, when the max-age expires, a If-Not-Modified must not return true after expiry even if the database entry of the wiki page was unchanged.
    • More info: https://phabricator.wikimedia.org/T46570
  • 2016: Decrease max object ttl in Varnish