You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Uploading large files: Difference between revisions
imported>Legoktm (include --sleep in example) |
imported>Quiddity m (fixes) |
||
| Line 11: | Line 11: | ||
Download the files to [[mwmaint1002]] (or if there's not enough space, deploy1001). | Download the files to [[mwmaint1002]] (or if there's not enough space, deploy1001). | ||
< | <syntaxhighlight lang="bash"> | ||
wget <URL> | wget <URL> | ||
</ | </syntaxhighlight> | ||
At this stage, it could be pertinent to check the hash of the file if known. | At this stage, it could be pertinent to check the hash of the file if known. | ||
| Line 19: | Line 19: | ||
== Step 2: import image to Commons == | == Step 2: import image to Commons == | ||
Server-side uploads run much faster because of minimal network overhead, and as a result can cause extra strain on the job queue, especially videos which require transcoding. It's recommended to add some delay in between each upload with the <code>--sleep</code> parameter. Because videos have various factors (resolution, fps, length) that would affect how long transcodes would take, it might be worth uploading one video, seeing how long the median transcode takes, and then sleeping for that length to avoid queuing up a large number of transcodes. | Server-side uploads run much faster because of minimal network overhead, and as a result can cause extra strain on the job queue, especially videos which require transcoding. It's recommended to add some delay in between each upload with the <code>--sleep</code> parameter. Because videos have various factors (resolution, fps, length) that would affect how long transcodes would take, it might be worth uploading one video, seeing how long the median transcode takes, and then sleeping for that length to avoid queuing up a large number of transcodes. | ||
< | <syntaxhighlight lang="bash"> | ||
mwscript importImages.php --wiki=commonswiki --sleep=SECONDS --comment-ext=txt --user=USERNAME /tmp/uploads | mwscript importImages.php --wiki=commonswiki --sleep=SECONDS --comment-ext=txt --user=USERNAME /tmp/uploads | ||
</ | </syntaxhighlight> | ||
Revision as of 19:18, 4 September 2021
Requirement
You need:
- the URL of the media file to upload
- a text file with the first revision content
- the name of the user account for this first revision and upload
MediaWiki currently doesn't support files greater than 4 GB (as size is stored as a 32 bits unsigned integer) while our swift backend storage is limited to 5 Gb. See phab:T191804 and phab:T191802 for discussion to extend this limit respectively to 5 GB and beyond.
Step 1: download files
Download the files to mwmaint1002 (or if there's not enough space, deploy1001).
wget <URL>
At this stage, it could be pertinent to check the hash of the file if known.
Step 2: import image to Commons
Server-side uploads run much faster because of minimal network overhead, and as a result can cause extra strain on the job queue, especially videos which require transcoding. It's recommended to add some delay in between each upload with the --sleep parameter. Because videos have various factors (resolution, fps, length) that would affect how long transcodes would take, it might be worth uploading one video, seeing how long the median transcode takes, and then sleeping for that length to avoid queuing up a large number of transcodes.
mwscript importImages.php --wiki=commonswiki --sleep=SECONDS --comment-ext=txt --user=USERNAME /tmp/uploads