You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Thumbor: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Filippo Giunchedi
No edit summary
imported>Filippo Giunchedi
(add Debian package instructions)
Line 18: Line 18:
=== Purging ===
=== Purging ===
Thumbor is currently stateless and doesn't need to concern itself with purges.
Thumbor is currently stateless and doesn't need to concern itself with purges.
== Operations ==
=== Debian package ===
Thumbor is deployed via Debian packages, specifically <code>python-thumbor-wikimedia</code> contains WMF extensions to process additional file types, talk to swift and so on. The repository with <code>debian/</code> directory lives at <code>operations/debs/python-thumbor-wikimedia</code> while "upstream" repository is at <code>https://phabricator.wikimedia.org/diffusion/THMBREXT/</code>.
Assuming <code>debian/changelog</code> has been updated, it is possible to build a new package by first tagging <code>upstream/VERSION</code> the relevant commit and then <code>git-buildpackage -us -uc -S</code> to create a <code>.orig.tar.gz</code> and <code>dpkg-source -b .</code> to create the source package. Once the source package is available it can be built with <code>BACKPORTS=yes DIST=jessie-wikimedia  sudo -E cowbuilder --debbuildopts -sa --build DSC_FILE</code> and uploaded to <code>apt.wikimedia.org</code>

Revision as of 08:51, 13 October 2016

There is an ongoing effort to rewrite our production thumbnailing infrastructure around Thumbor servers. The objective is to ultimately stop storing thumbnails in Swift, to remove the need to run mediawiki on image scalers and to generally have a more maintainable platform.

This new Thumbor service is meant to be stateless and only concern itself with transforming images.

Currently the new Thumbor-based stack is being developed on Vagrant. Also deployed in production in eqiad (thumbor1001.eqiad.wmnet, thumbor1002.eqiad.wmnet), fetching originals from swift and storing thumbnails back into swift with the thumbor/ prefix.

After that, we will need to figure out a replacement for Swift, in all likelihood an LRU cache that will run on its own cluster and won't store more than one copy of each thumbnail. Thumbor will be pointed to that instead of Swift to store thumbnails and we can decommission the storage of thumbnails in Swift.

File types

Adapters for all the file formats used on Commons have been written for Thumbor (JPG, PNG, GIF, TIFF, XCF, SVG, PDF, DJVU, WEBM, OGV).

Originals

Thumbor reads its originals directly from Swift and only caches them in memory for the duration of the thumbnail's request, in case a concurrent request on the same Thumbor server wants the same original (which is particularly useful in prerendering/warming situations). After the request is done, it doesn't keep any copy of the original anywhere.

Thumbnails

Thumbor stores thumbnails on Swift, using the same location scheme as image scalers. It doesn't cache nor store thumbnails locally.

Purging

Thumbor is currently stateless and doesn't need to concern itself with purges.

Operations

Debian package

Thumbor is deployed via Debian packages, specifically python-thumbor-wikimedia contains WMF extensions to process additional file types, talk to swift and so on. The repository with debian/ directory lives at operations/debs/python-thumbor-wikimedia while "upstream" repository is at https://phabricator.wikimedia.org/diffusion/THMBREXT/.

Assuming debian/changelog has been updated, it is possible to build a new package by first tagging upstream/VERSION the relevant commit and then git-buildpackage -us -uc -S to create a .orig.tar.gz and dpkg-source -b . to create the source package. Once the source package is available it can be built with BACKPORTS=yes DIST=jessie-wikimedia sudo -E cowbuilder --debbuildopts -sa --build DSC_FILE and uploaded to apt.wikimedia.org