You are browsing a read-only backup copy of Wikitech. The live site can be found at


From Wikitech-static
Revision as of 04:13, 21 April 2015 by imported>GWicke (→‎HOWTO)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Varnish is a fast caching proxy, and can be used as an alternative to Squid in a reverse caching accelerator setup.

We currently use Varnish for serving,, text of pages retrieved from WMF projects, and various miscellaneous web services. Nothing uses Squid.


See Varnish statistics


# varnishstat

Fred has also written a Ganglia plugin in Python for varnish, which is automatically installed by Puppet. All varnishstat metrics are therefore visible on Ganglia.

Set runtime parameters


# varnishadm -S /etc/varnish/secret -T

Add/remove backends

Puppet can automatically generate Varnish backend and director statements by setting the $varnish_backends and $varnish_directors variables (see below).

Alternatively, backends can be defined manually in the templates/varnish/wikimedia.vcl.erb ERB template file.

See request logs

As explained below, there are no access logs. However, you can see NCSA style log entries for current requests using:

# varnishncsa

See backend health


# varnishlog -i Backend_health -O

Some more tricks

// Query Times
varnishncsa -F '%t %{VCL_Log:Backend}x %Dμs %bB %s %{Varnish:hitmiss}x "%r"'

// Top URLs
varnishtop -i RxURL

// Top Referer, User-Agent, etc.
varnishtop -i RxHeader -I Referer
varnishtop -i RxHeader -I User-Agent

// Cache Misses
varnishtop -i TxURL


Deployment of Varnish is done using Puppet, using class varnish in file manifests/varnish.pp.

We use the varnish package of Ubuntu/Debian, minimum version 2.1.2. Puppet installs this, and replaces its /etc/default/varnish file to set some startup parameters (discussed below). Varnish uses a VCL file (Varnish Configuration Language), a DSL where Varnish behaviour is controlled using subroutines that are compiled into C and executed during each request. The Wikimedia VCL file is


Like with Squid, special sysctl settings are installed by Puppet to tune the system for high HTTP traffic performance.

Varnish does not use log files, but instead writes detailed information about its operations to a SHM ring buffer of a fixed size. Any interested programs can just read along and produce statistics or log output without it slowing down the Varnish daemon itself. The SHM file is mlocked in memory, but Linux insists on writing its buffers to disk anyway - therefore Puppet mounts the directory /var/lib/ganglia into a 150M sized tmpfs filesystem to avoid this.

Startup parameters

The following parameters have been changed from the defaults. Most of these are set in the (now Puppet generated) file /etc/default/varnish.

  • NFILES=500000

For ulimit -n, the number of files/sockets/file descriptors Varnish can open

  • MEMLOCK=90000

For ulimit -l, so Varnish can lock the entire SHM log buffer (default 80M) into memory.

  • ulimit -s 128

To reduce the VSIZE with many Varnish threads

  • VARNISH_VCL_CONF=/etc/varnish/wikimedia.vcl

Points to the Wikimedia specific VCL file


Bind to TCP port 80 on all IPs


Listening socket for the administrative interface


The minimum and maximum amounts of threads Varnish will keep around for requests, per thread pool.

  • VARNISH_STORAGE="malloc,1G"

For bits, which has a small content set, we want to keep everything in memory.

  • EXTRA_OPTS="-p thread_pools=8 -p thread_pool_add_delay=1 -p send_timeout=30 -p listen_depth=4096"

Extra runtime parameters, explained below:

  • thread_pools=8

One thread pool per CPU core; this reduces mutex contention.

  • thread_pool_add_delay

Create more threads quickly when needed

  • send_timeout=3

Keep the amount of open connections low, and close idle connections quickly.

  • listen_depth=4096

Allow many new connections in the accept() queue, before Varnish can open them.

Puppet configuration

In Puppet, two variables can be defined:

an array of fully qualified hostnames that will be used to generate Varnish backends in the VCL, e.g. $varnish_backends = [ "srv191.pmtpa.wmnet", "srv192.pmtpa.wmnet" ]
a hash of director names to backends, e.g. $varnish_directors = { "appservers" => [ "srv191.pmtpa.wmnet", "srv192.pmtpa.wmnet" ] }

You also may want to read Bits varnish testing instead, for Domas his findings during a pilot project.

One-off purges

$ dsh -c -g bits varnishadm -T -S /etc/varnish/secret ban.url <your url here>

Don't do this. Consult a varnish specialist first.

< bblack> (it's generally a varnishadm ban command, but the syntax/caveats are not exactly intuitive)
< bblack> as in, figure out the right ban command in , then push that around to the correct cluster of caches via salt
< bblack> the commands I did were:
< bblack> root@palladium:~# salt -G 'cluster:cache_text' 'varnishadm -S /etc/varnish/secret -T ban == ""'
< bblack> and then repeat, but change :6083 to :6082 (to hit frontends after fixing on backends)
< bblack> and variations of course: cluster:cache_* or cluster:cache_mobile, etc... and the ban expression can be complex, a lot of the same things available in req/resp objects in VCL in general 


When mobile wants you to 'clear the varnish cache', you should read MobileFrontend#Flushing_the_cache instead of this page.

Things that need special consideration

  • HTCP purging
  • Immediate purging of cache objects (nuke?)
  • Header normalization (Host, Accept-Encoding...)
  • Two-layer setup (CARP style)
  • Compatible logging
  • Request stats

Would be nice

  • SSL
  • IPv6

Many of these are probably already taken care of by our friends at Wikia, and therefore possibly also within Varnish itself...

External links