Thumbor

As of June 2017, the Wikimedia production (and beta) media thumbnailing infrastructure is based on Thumbor. All thumbnail traffic for public wikis is served by it.

Migrating private WMF wikis to Thumbor is the subject of a follow-up project.

Rationale

• Better support Thumbor has a lively community of its own, and is a healthy open-source project. In contrast, the media-handling code in Mediawiki is supported on a best-effort basis by very few people
• Better security isolation Thumbor is stateless and connects to Swift, Poolcounter and DC-local Thumbor-specific Memcache instances (see "Throttling" below). In contrast, Mediawiki is connected to many more services, as well as user data and sessions. Considering how common security vulnerability discoveries are in media-processing software, it makes sense to isolate media thumbnailing as much as possible.
• Better separation of concerns Thumbor only concerns itself with thumbnail generation. This is desirable in a service-oriented architecture.
• Easier operations Thumbor is a simple service and should be easy to operate.

Supported file types

We have written Thumbor engines for all the file formats used on Wikimedia wikis (JPEG, PNG, GIF, TIFF, XCF, SVG, PDF, DJVU, WEBM, OGV, WEBM, STL).

These engines reuse the same logic as Mediawiki to render those images, often leveraging the same underlying open-source libraries or executables. Whenever possible, reference images generated with Mediawiki are used for the Thumbor integration tests.

Broader ecosystem

In order to understand Thumbor's role in our software stack, one has to understand how Wikimedia production is currently serving those images.

The edge, where user requests first land, is Varnish. Most requests for a thumbnail are a hit on the Varnish frontend or backend caches.

When Varnish can't find a copy of the requested thumbnail - whether it's a thumbnail that has never been requested before, or ones that fell out of Varnish cache - Varnish hits the Swift proxies. We run a custom plugin on our Swift proxies, which is responsible for parsing the thumbnail URL, determining whether there is a copy of that thumbnail already stored in Swift, serving it if that's the case, asking Thumbor to generate it otherwise.

When Thumbor receives a request, it tries to fetch the original media from Swift. If it can't, it 404s. It then proceeds to generate the request thumbnail for that media. Once it's done, it serves the resulting image, which the Swift proxy then forwards to Varnish, which serves it to the client. Varnish saves a copy in its own cache, and Thumbor saves a copy in Swift.

Throttling

In order to prevent abuse and to distribute server resources more fairly, Thumbor has a few throttling mechanisms in place. These happen as early as possible in the request handling, in order to avoid unnecessary work.

Memcached-based

Failure throttling require having a memory of past events. For this we use Memcached. In order to share the throttling information across Thumbor instances, we use a local nutcracker instance running on each Thumbor server, pointing to all the Thumbor servers in a given datacenter. This is configured in Puppet, with the list of servers in hiera under the thumbor_memcached_servers and thumbor_memcached_servers_nutcracker config variables.

In Thumbor's configuration, the memcached settings used for this are defined in FAILURE_THROTTLING_MEMCACHE and FAILURE_THROTTLING_PREFIX, found in Puppet.

Failure

The failure throttling logic itself is governed by the FAILURE_THROTTLING_MAX and FAILURE_THROTTLING_DURATION Thumbor config variables. This throttling limits retries on failing thumbnails. Some originals are broken or can't be rendered by our thumbnailing software and there would be no point retrying them every time we encounter them. This limit allows us to avoid rendering problematic originals for a while. We don't want to blacklist them permanently, however, as upgrading media-handling software might suddenly make originals that previously couldn't be rendered start working. This limit having an expiry guarantees that the benefits of upgrades apply naturally to problematic files, without requiring to clear a permanent blacklist whenever software is upgraded on the Thumbor hosts.

Poolcounter-based

For other forms of throttling, we use Poolcounter. Both to combat malicious of unintentional DDoS, as well as regulate resource consumption. The Poolcounter server configuration shared by the different throttling types is defined in the POOLCOUNTER_SERVER, POOLCOUNTER_PORT and POOLCOUNTER_RELEASE_TIMEOUT Thumbor config variables, found in Puppet.

Per-IP

We limit the amount of concurrent thumbnail generation requests per client IP address. The configuration for that throttle is governed by the and POOLCOUNTER_CONFIG_PER_IP Thumbor config variable, found in Puppet.

Per-original

We limit the amount of concurrent thumbnail generation requests per original media. The configuration for that throttle is governed by the and POOLCOUNTER_CONFIG_PER_ORIGINAL Thumbor config variable, found in Puppet.

Expensive

Some thumbnail types are disproportionately expensive to render thumbnails for (in terms of CPU time, mostly). Those expensive types are subject to an extra throttle, defined by the POOLCOUNTER_CONFIG_EXPENSIVE Thumbor config variable, found in Puppet.

Not per-user

Unlike Mediawiki, Thumbor doesn't implement a per-user Poolcounter throttle. First because Thumbor has greater isolation (on purpose) and doesn't have access to any user data, including sessions. Secondly, the per-IP throttle should covers the same ground, as logged-in users should have little IP address variance during a session.

Logging

Thumbor logs go to /srv/log/thumbor on the Thumbor servers. All the Thumbor instances on a given server write to the same files. Logs are rotated daily. The logging configuration is defined in Puppet, under the THUMBOR_LOG_CONFIG Thumbor config variable.

Thumbor logs also go to Logstash/Kibana.

Configuration

Thumbor consumes its configuration from the /etc/thumbor.d/ folder. The .conf files found in that folder are parsed in alphabetical order by Thumbor. The thumbor Debian package as well as our custom python-thumbor-wikimedia Debian package contain default configuration files. On top of which we add some defined in Puppet.

The rule of thumb here is that configuration that might depend on the instance or datacenter at hand should be defined in Puppet, while configuration that won't vary per machine can be defined in the python-thumbor-wikimedia Debian package.

Operations

Dashboards

Grafana thumbor dashboard

Thumbor Prometheus Eqiad breakdown

Restarting

 cumin -b1 -s10 'thumbor1*' 'systemctl restart thumbor-instances'
 cumin -b1 -s10 'thumbor2*' 'systemctl restart thumbor-instances'

Debian package

Thumbor is deployed via Debian packages, specifically python-thumbor-wikimedia contains WMF extensions to process additional file types, talk to Swift and so on. The repository with debian/ directory lives at operations/debs/python-thumbor-wikimedia while "upstream" repository is at https://phabricator.wikimedia.org/diffusion/THMBREXT/.

Assuming debian/changelog has been updated, it is possible to build a new package by first tagging upstream/VERSION the relevant commit and then git-buildpackage -us -uc -S to create a .orig.tar.gz and dpkg-source -b . to create the source package. Once the source package is available it can be built with BACKPORTS=yes DIST=jessie-wikimedia sudo -E cowbuilder --debbuildopts -sa --build DSC_FILE and uploaded to apt.wikimedia.org

Using manhole

Thumbor runs with python manhole for debugging/inspection purposes. See also T146143: Figure out a way to live-debug running production thumbor processes

To invoke manhole, e.g. on thumbor on port 8827:

 sudo -u thumbor socat - unix-connect:/srv/thumbor/tmp/thumbor@8827/manhole-8827