You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Thumbor: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Gilles
No edit summary
 
imported>Gilles
No edit summary
Line 1: Line 1:
There is an [[phab:T111718|ongoing effort]] to rewrite our production thumbnailing infrastructure around [http://thumbor.org/ Thumbor] servers. The objective is to ultimately stop storing thumbnails in Swift, to remove the need to run mediawiki on image scalers and to generally have a more maintainable platform.
There is an [[phab:T111718|ongoing effort]] to rewrite our production thumbnailing infrastructure around [http://thumbor.org/ Thumbor] servers. The objective is to ultimately stop storing thumbnails in Swift, to remove the need to run mediawiki on image scalers and to generally have a more maintainable platform.


At this stage the new Thumbor-based stack is being developed on Vagrant. The working proof-of-concept developed that way will serve as the basis of the upcoming RfC to deploy this solution to production in stages.
This new Thumbor service is meant to be stateless and only concern itself with transforming images.
 
Currently the new Thumbor-based stack is being developed on Vagrant.
 
The next stage will be to deploy Thumbor on labs and production while aiming for it to be a drop-in replacement for image scalers. At that point Thumbor will still write thumbnails to Swift.
 
After that, we will need to figure out a replacement for Swift, in all likelihood an LRU cache that will run on its own cluster and won't store more than one copy of each thumbnail. Thumbor will be pointed to that instead of Swift to store thumbnails and we can decommission the storage of thumbnails in Swift.


=== File types ===
=== File types ===
Currently Thumbor is only configured to handle JPG and PNG files. Support for other file formats will follow, meanwhile those other formats will still rely on the existing thumbnailing infrastructure.
Adapters for all the file formats used on Commons have been written for Thumbor (JPG, PNG, GIF, TIFF, XCF, SVG, PDF, DJVU, WEBM, OGV).


=== Originals ===
=== Originals ===
Thumbor is configured to keep a copy of original images in memcache for a short period of time. This is to cater for burst requests of several thumbnails of the same file, which happens during normal operations at upload time, when we pre-render the most popular thumbnail sizes.
Thumbor reads its originals directly from Swift and only caches them in memory for the duration of the thumbnail's request, in case a concurrent request on the same Thumbor server wants the same original (which is particularly useful in prerendering/warming situations). After the request is done, it doesn't keep any copy of the original anywhere.


=== '''Thumbnails''' ===
=== Thumbnails ===
Thumbor stores thumbnails on disk. This is only useful for disaster scenarios where Varnish would fail partly or fully. The plan to avoid the disk getting full is to delete thumbnails randomly when disk usage goes above a certain threshold. The randomness of the deletion is due to Thumbor having no way to know the traffic patterns of thumbnails when Varnish is in front of it. Deleting by age would be counter-productive, as hot thumbnails in Varnish are more likely to get old on Thumbor servers.
Thumbor stores thumbnails on Swift, using the same location scheme as image scalers. It doesn't cache nor store thumbnails locally.


=== '''Purging''' ===
=== Purging ===
Thumbor has a single purging endpoint, which deletes a given original file and all its thumbnails from the Thumbor server. In situations of deletions (eg. copyvios), one should make sure that the original file is deleted in Swift first. The order of deletion/purging should be swift -> thumbor -> varnish backends -> varnish frontends.
Thumbor is currently stateless and doesn't need to concern itself with purges.

Revision as of 14:10, 8 February 2016

There is an ongoing effort to rewrite our production thumbnailing infrastructure around Thumbor servers. The objective is to ultimately stop storing thumbnails in Swift, to remove the need to run mediawiki on image scalers and to generally have a more maintainable platform.

This new Thumbor service is meant to be stateless and only concern itself with transforming images.

Currently the new Thumbor-based stack is being developed on Vagrant.

The next stage will be to deploy Thumbor on labs and production while aiming for it to be a drop-in replacement for image scalers. At that point Thumbor will still write thumbnails to Swift.

After that, we will need to figure out a replacement for Swift, in all likelihood an LRU cache that will run on its own cluster and won't store more than one copy of each thumbnail. Thumbor will be pointed to that instead of Swift to store thumbnails and we can decommission the storage of thumbnails in Swift.

File types

Adapters for all the file formats used on Commons have been written for Thumbor (JPG, PNG, GIF, TIFF, XCF, SVG, PDF, DJVU, WEBM, OGV).

Originals

Thumbor reads its originals directly from Swift and only caches them in memory for the duration of the thumbnail's request, in case a concurrent request on the same Thumbor server wants the same original (which is particularly useful in prerendering/warming situations). After the request is done, it doesn't keep any copy of the original anywhere.

Thumbnails

Thumbor stores thumbnails on Swift, using the same location scheme as image scalers. It doesn't cache nor store thumbnails locally.

Purging

Thumbor is currently stateless and doesn't need to concern itself with purges.