You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Video/Help/video2commons: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Zhuyifei1999
(→‎Setup environment: config.py => json)
imported>Quiddity
m (fixes)
 
(2 intermediate revisions by 2 users not shown)
Line 18: Line 18:


== Web frontend ==
== Web frontend ==
Webfrontend is on [[Nova resource:Tools|Tool labs]] under tool [http://tools.wmflabs.org/?tool=video2commons video2commons].
Webfrontend is on [[Nova resource:Tools|Tool labs]] under tool [https://video2commons.toolforge.org/ video2commons].


=== Setup ===
=== Setup ===
Ensure you're in your tool account; if not, <source lang="bash" enclose="none">$ become video2commons</source>
Ensure you're in your tool account; if not, <syntaxhighlight lang="bash" inline>$ become video2commons</syntaxhighlight>


=== Clone repo ===
=== Clone repo ===
Somehow, clone [https://github.com/Toollabs/video2commons.git video2commons] to the tool's home dir. Some methods can do this:
Somehow, clone [https://github.com/Toollabs/video2commons.git video2commons] to the tool's home dir. Some methods can do this:
* Delete everything and clone it (be careful): <source lang="bash" enclose="none">$ rm -rf * && git clone https://github.com/Toollabs/video2commons.git .</source>
* Delete everything and clone it (be careful): <syntaxhighlight lang="bash" inline>$ rm -rf * && git clone https://github.com/Toollabs/video2commons.git .</syntaxhighlight>
* Or, clone to a <code>tmp</code> dir and copy git configuration back: <source lang="bash" enclose="none">$ git clone https://github.com/Toollabs/video2commons.git tmp && mv tmp/.git . && rm -rf tmp && git reset --hard</source>
* Or, clone to a <code>tmp</code> dir and copy git configuration back: <syntaxhighlight lang="bash" inline>$ git clone https://github.com/Toollabs/video2commons.git tmp && mv tmp/.git . && rm -rf tmp && git reset --hard</syntaxhighlight>


=== Setup environment ===
=== Setup environment ===
Inside a shell of the tool's k8s container (launch with <source lang="bash" enclose="none">$ webservice --backend=kubernetes python2 shell</source>):
Inside a shell of the tool's k8s container (launch with <syntaxhighlight lang="bash" inline>$ webservice --backend=kubernetes python2 shell</syntaxhighlight>):
Setup virtualenv:
Setup virtualenv:
<source lang="bash">
<syntaxhighlight lang="bash">
$ virtualenv ~/www/python/venv
$ virtualenv ~/www/python/venv
$ echo source ~/www/python/venv/bin/activate >> .bash_profile
$ echo source ~/www/python/venv/bin/activate >> .bash_profile
Line 39: Line 39:


Install necessary components:
Install necessary components:
* <source lang="bash" enclose="none">$ pip install -r requirements.txt</source>
* <syntaxhighlight lang="bash" inline>$ pip install -r requirements.txt</syntaxhighlight>
* Anything else?
* Anything else?


Line 46: Line 46:
=== Start the webservice ===
=== Start the webservice ===
Start the frontend (outside of the container):
Start the frontend (outside of the container):
<source lang="bash">
<syntaxhighlight lang="bash">
$ webservice --backend=kubernetes python2 start
$ webservice --backend=kubernetes python2 start
</source>
</syntaxhighlight>


=== Updating ===
=== Updating ===
Run <source lang="bash" enclose="none">$ git pull</source>. No local commits should exist.
Run <syntaxhighlight lang="bash" inline>$ git pull</syntaxhighlight>. No local commits should exist.


* If any python code is changed, uwsgi must be restarted: <source lang="bash" enclose="none">$ webservice --backend=kubernetes python2 restart</source> (and a downtime under a minute might be caused)
* If any python code is changed, uwsgi must be restarted: <syntaxhighlight lang="bash" inline>$ webservice --backend=kubernetes python2 restart</syntaxhighlight> (and a downtime under a minute might be caused)
* Otherwise, nothing must be done.
* Otherwise, nothing must be done.


Line 63: Line 63:


Use puppet to configure the rest:
Use puppet to configure the rest:
# Get the puppet configuration: <source lang="bash" enclose="none">$ wget https://github.com/Toollabs/video2commons/raw/master/puppet/backend.pp</source>
# Get the puppet configuration: <syntaxhighlight lang="bash" inline>$ wget https://github.com/Toollabs/video2commons/raw/master/puppet/backend.pp</syntaxhighlight>
# Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil.
# Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil.
# Apply them: <source lang="bash" enclose="none">$ sudo puppet apply backend.pp --debug</source>
# Apply them: <syntaxhighlight lang="bash" inline>$ sudo puppet apply backend.pp --debug</syntaxhighlight>


=== Switching off ===
=== Switching off ===


Run once (and only once. DO NOT run a second time unless something is already terribly broken!):
Run once (and only once. DO NOT run a second time unless something is already terribly broken!):
<source lang="bash">
<syntaxhighlight lang="bash">
sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2
sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2
</source>
</syntaxhighlight>
This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers.
This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers.

Latest revision as of 18:34, 4 September 2021

Toolforge tools
Crystal Clear app package utilities.png video2commons
Video2commons screenshot.jpg
Website toollabs:video2commons
Description Transfers video and audio from external sites to Commons
Keywords video, transfer
Author(s) Zhuyifei1999talk
Maintainer(s) Zhuyifei1999 (View all)
Source code GitHub
License GNU General Public License 3.0 or later
Issues GitHub

This tool transfers video and audio from external sites to Commons. Web frontend is on Nova resource:Tools, and backend is on Nova resource:Video. Code is hosted on GitHub.

Web frontend

Webfrontend is on Tool labs under tool video2commons.

Setup

Ensure you're in your tool account; if not, $ become video2commons

Clone repo

Somehow, clone video2commons to the tool's home dir. Some methods can do this:

  • Delete everything and clone it (be careful): $ rm -rf * && git clone https://github.com/Toollabs/video2commons.git .
  • Or, clone to a tmp dir and copy git configuration back: $ git clone https://github.com/Toollabs/video2commons.git tmp && mv tmp/.git . && rm -rf tmp && git reset --hard

Setup environment

Inside a shell of the tool's k8s container (launch with $ webservice --backend=kubernetes python2 shell): Setup virtualenv:

$ virtualenv ~/www/python/venv
$ echo source ~/www/python/venv/bin/activate >> .bash_profile
$ source ~/www/python/venv/bin/activate
$ pip install -U pip
</source>

Install necessary components:
* <syntaxhighlight lang="bash" inline>$ pip install -r requirements.txt
  • Anything else?

Create a config.json similar to config.json.example, and fill the necessary configs. Make sure the file is chmodded 600 before writing it!

Start the webservice

Start the frontend (outside of the container):

$ webservice --backend=kubernetes python2 start

Updating

Run $ git pull. No local commits should exist.

  • If any python code is changed, uwsgi must be restarted: $ webservice --backend=kubernetes python2 restart (and a downtime under a minute might be caused)
  • Otherwise, nothing must be done.

Backend

Backends are celery workers on video cluster encoding instances.

Setup

Ensure Nova Resource:Video/Help#Setup is done.

Use puppet to configure the rest:

  1. Get the puppet configuration: $ wget https://github.com/Toollabs/video2commons/raw/master/puppet/backend.pp
  2. Ensure the first few lines of the file is correct. And, just in case, make sure nothing in the code is evil.
  3. Apply them: $ sudo puppet apply backend.pp --debug

Switching off

Run once (and only once. DO NOT run a second time unless something is already terribly broken!):

sudo celery -A video2commons.backend.worker --workdir=/srv/v2c multi stop --pidfile="/var/run/v2ccelery/%N.pid" 2

This will put celery to gracefully shutdown (i.e. it will stop handling new jobs, stop all idle workers, and wait for each worker to finish its job). Running a second time WILL KILL all the workers.