You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Nova Resource:Video/Help/video2commons: Difference between revisions
< Nova Resource:Video | Help
Jump to navigation
Jump to search
imported>Steinsplitter mNo edit summary |
imported>Quiddity m (fixes) |
||
| (One intermediate revision by one other user not shown) | |||
| Line 18: | Line 18: | ||
== Web frontend == | == Web frontend == | ||
Webfrontend is on [[Nova resource:Tools|Tool labs]] under tool [https:// | Webfrontend is on [[Nova resource:Tools|Tool labs]] under tool [https://video2commons.toolforge.org/ video2commons]. | ||
=== Setup === | === Setup === | ||
Ensure you're in your tool account; if not, < | Ensure you're in your tool account; if not, <syntaxhighlight lang="bash" inline>$ become video2commons</syntaxhighlight> | ||
=== Clone repo === | === Clone repo === | ||
Somehow, clone [https://github.com/Toollabs/video2commons.git video2commons] to the tool's home dir. Some methods can do this: | Somehow, clone [https://github.com/Toollabs/video2commons.git video2commons] to the tool's home dir. Some methods can do this: | ||
* Delete everything and clone it (be careful): < | * Delete everything and clone it (be careful): <syntaxhighlight lang="bash" inline>$ rm -rf * && git clone https://github.com/Toollabs/video2commons.git .</syntaxhighlight> | ||
* Or, clone to a <code>tmp</code> dir and copy git configuration back: < | * Or, clone to a <code>tmp</code> dir and copy git configuration back: <syntaxhighlight lang="bash" inline>$ git clone https://github.com/Toollabs/video2commons.git tmp && mv tmp/.git . && rm -rf tmp && git reset --hard</syntaxhighlight> | ||
=== Setup environment === | === Setup environment === | ||
Inside a shell of the tool's k8s container (launch with < | Inside a shell of the tool's k8s container (launch with <syntaxhighlight lang="bash" inline>$ webservice --backend=kubernetes python2 shell</syntaxhighlight>): | ||
Setup virtualenv: | Setup virtualenv: | ||
< | <syntaxhighlight lang="bash"> | ||
$ virtualenv ~/www/python/venv | $ virtualenv ~/www/python/venv | ||
$ echo source ~/www/python/venv/bin/activate >> .bash_profile | $ echo source ~/www/python/venv/bin/activate >> .bash_profile | ||
| Line 39: | Line 39: | ||
Install necessary components: | Install necessary components: | ||
* < | * <syntaxhighlight lang="bash" inline>$ pip install -r requirements.txt</syntaxhighlight> | ||
* Anything else? | * Anything else? | ||
| Line 46: | Line 46: | ||
=== Start the webservice === | === Start the webservice === | ||
Start the frontend (outside of the container): | Start the frontend (outside of the container): | ||
< | <syntaxhighlight lang="bash"> | ||
$ webservice --backend=kubernetes python2 start | $ webservice --backend=kubernetes python2 start | ||
</ | </syntaxhighlight> | ||
=== Updating === | === Updating === | ||
Run < | Run <syntaxhighlight lang="bash" inline>$ git pull</syntaxhighlight>. No local commits should exist. | ||
* If any python code is changed, uwsgi must be restarted: < | * If any python code is changed, uwsgi must be restarted: <syntaxhighlight lang="bash" inline>$ webservice --backend=kubernetes python2 restart</syntaxhighlight> (and a downtime under a minute might be caused) | ||
* Otherwise, nothing must be done. | * Otherwise, nothing must be done. | ||
| Line 63: | Line 63: | ||
Use puppet to configure the rest: | Use puppet to configure the rest: | ||
# Get the puppet configuration: < | # Get the puppet configuration: <syntaxhighlight lang="bash" inline>$ wget https://github.com/Toollabs/video2commons/raw/master/puppet/backend.pp</syntaxhighlight> | ||
# Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil. | # Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil. | ||
# Apply them: < | # Apply them: <syntaxhighlight lang="bash" inline>$ sudo puppet apply backend.pp --debug</syntaxhighlight> | ||
=== Switching off === | === Switching off === | ||
Run once (and only once. DO NOT run a second time unless something is already terribly broken!): | Run once (and only once. DO NOT run a second time unless something is already terribly broken!): | ||
< | <syntaxhighlight lang="bash"> | ||
sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2 | sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2 | ||
</ | </syntaxhighlight> | ||
This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers. | This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers. | ||
Latest revision as of 18:34, 4 September 2021
 video2commons
| |
|---|---|
| |
| Website | toollabs:video2commons |
| Description | Transfers video and audio from external sites to Commons |
| Keywords | video, transfer |
| Author(s) | Zhuyifei1999talk |
| Maintainer(s) | Zhuyifei1999 (View all) |
| Source code | GitHub |
| License | GNU General Public License 3.0 or later |
| Issues | GitHub |
This tool transfers video and audio from external sites to Commons. Web frontend is on Nova resource:Tools, and backend is on Nova resource:Video. Code is hosted on GitHub.
Web frontend
Webfrontend is on Tool labs under tool video2commons.
Setup
Ensure you're in your tool account; if not, $ become video2commons
Clone repo
Somehow, clone video2commons to the tool's home dir. Some methods can do this:
- Delete everything and clone it (be careful):
$ rm -rf * && git clone https://github.com/Toollabs/video2commons.git . - Or, clone to a
tmpdir and copy git configuration back:$ git clone https://github.com/Toollabs/video2commons.git tmp && mv tmp/.git . && rm -rf tmp && git reset --hard
Setup environment
Inside a shell of the tool's k8s container (launch with $ webservice --backend=kubernetes python2 shell):
Setup virtualenv:
$ virtualenv ~/www/python/venv
$ echo source ~/www/python/venv/bin/activate >> .bash_profile
$ source ~/www/python/venv/bin/activate
$ pip install -U pip
</source>
Install necessary components:
* <syntaxhighlight lang="bash" inline>$ pip install -r requirements.txt
- Anything else?
Create a config.json similar to config.json.example, and fill the necessary configs. Make sure the file is chmodded 600 before writing it!
Start the webservice
Start the frontend (outside of the container):
$ webservice --backend=kubernetes python2 start
Updating
Run $ git pull. No local commits should exist.
- If any python code is changed, uwsgi must be restarted:
$ webservice --backend=kubernetes python2 restart(and a downtime under a minute might be caused) - Otherwise, nothing must be done.
Backend
Backends are celery workers on video cluster encoding instances.
Setup
Ensure Nova Resource:Video/Help#Setup is done.
Use puppet to configure the rest:
- Get the puppet configuration:
$ wget https://github.com/Toollabs/video2commons/raw/master/puppet/backend.pp - Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil.
- Apply them:
$ sudo puppet apply backend.pp --debug
Switching off
Run once (and only once. DO NOT run a second time unless something is already terribly broken!):
sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2
This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers.