You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Ores/Deployment: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Alexandros Kosiaris
No edit summary
imported>Ladsgroup
(Ladsgroup moved page Ores/Deployment to ORES/Deployment)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
This page is a guide on how to deploy new version of ORES into the server.
#REDIRECT [[ORES/Deployment]]
== PyPI ==
So, your patches are merged into ores/revscoring/other dependencies. You need to increment the version number. Try to do that in a [http://semver.org/ SemVer] fashion. Like only upgrading the patch level (e.g. 0.5.8 -> 0.5.9). You need to do it in setup.py and __init__.py (and probably some other place too, use grep to check where the current version is used)
 
Then you need to push new version into PyPI using:
 
<pre>
python setup.py sdist bdist_wheel upload
</pre>
 
If you got GPG/PGP you can try adding sign to the list above to also sign the wheel and the sdist
 
== Update models ==
If you are doing breaking changes to revscoring probably old model files won't work, so you need to rebuild models. Do it using Makefile in <tt>editquality</tt> & <tt>wikiclass</tt> repos.  If a model changes substantially (new features, new algorithm, etc), make sure to increment the model versions in the Makefile too.
 
== Update wheels ==
First, clone https://github.com/wiki-ai/ores-wikimedia-config:
 
<pre>
git clone https://github.com/wiki-ai/ores-wikimedia-config
</pre>
 
There is a file in ores-wikimedia-config called "requirements.txt". Update their version number and make wheels by making a virtualenv and installing everything in it:
 
<pre>
virtualenv -p python3 tmp
source tmp/bin/activate
pip install wheel
pip wheel -w wheels/ -r requirements.txt
</pre>
 
It's better to do this in an environment similar to the production cluster. ores-compute-01.ores.eqiad.wmflabs and ores-compute-02.ores.eqiad.wmflabs are designed to do that. Don't forget to install C dependencies beforehand. Be careful if any kind of error happened.
 
Once wheels are ready, there is a repo in gerrit called wheels (in research/ores/wheels) we keep wheels and nltk data in it. You need to git clone, update wheels and make a patch:
 
<pre>
git clone ssh://YOURUSERNAME@gerrit.wikimedia.org:29418/research/ores/wheels
</pre>
 
Then, you need to copy new versions to wheels folder, '''delete old ones''' and make a new patch:
 
<pre>
cd wheels
git commit -m "New wheels for wiki-ai 1.2" -a
git review -R
</pre>
== Update ores-wikimedia-config ==
After +2ing and being merged, you should update ores-wikimedia-config
 
<pre>
cd ores-wikimedia-config
git checkout -b wiki_ai_1.2
source tmp/bin/activate
pip freeze | grep -v setuptools > requirements-frozen.txt
cd submodules/wheels
git pull
cd ../..
git commit -m "Release wiki-ai 1.2"
git push -f origin wiki_ai_1.2
</pre>
After that you need to make a PR in github and once it's merged it's good to go!
 
If you want to deploy to prod as well (ores.wikimedia.org) you need to backport your commits in gerrit too (ewww). The gerrit repos are:
<pre>
git clone ssh://YOURUSERNAME@gerrit.wikimedia.org:29418/mediawiki/services/ores
</pre>
For [https://github.com/wiki-ai/ores ores].
 
And:
* "mediawiki/services/ores/deploy" for [https://github.com/wiki-ai/ores-wikimedia-config ores-wikimedia-config] (note that these repos have diverged)
* "mediawiki/services/ores/editquality" for [https://github.com/wiki-ai/editquality editquality]
* "mediawiki/services/ores/wikiclass" for [https://github.com/wiki-ai/wikiclass wikiclass]
 
== Deploy ==
You need to login in deploy server to deploy new version using fabric or scap3
=== Labs (ores.wmflabs.org) ===
First, you need to go to staging. Simply make your changes in ores-wikimedia-config and do "fab stage" (don't forget to log it in #wikimedia-labs by typing this: "!log ores-staging deploying #hash into staging".
 
Then check ores-staging.wmflabs.org to see if everything is working alright and if it does, you are good to go to the labs setup.
First you need to rebase "deploy" branch.
<pre>
git checkout deploy
git rebase origin/master
git push -f origin deploy
</pre>
Then deploy with "fab deploy_web" and then "fab deploy_celery". Once it's done, test ores.wmflabs.org to see if everything is working as expected.
=== Beta (ores-beta.wmflabs.org) ===
Login to "deployment-tin.eqiad.wmflabs" (if you don't have permission to access to deployment-prep project, ask in #wikimedia-releng). Got to /srv/deployment/ores/deploy. Do your git magic (git pull and stuff) and then deploy via scap3 by commanding "scap deploy -v" and check out if anything works as expected. Log it in by "!log deploying ores #hash" in #wikimedia-releng
=== Production cluster (ores.wikimedia.org) ===
You are doing a dangerous thing. Remember, breaking the site is extremely easy! Be careful in every step and try to have someone from the team and ops supervising you. Also remember, ORES is depending on a huge number of puppet configurations, check out if your change is compatible with puppet configs and change puppet configs if necessary.
 
Deploy into "deployment.eqiad.wmnet". Then go to /srv/deployment/ores/deploy and then do git pull (and for submodules as well, if needed). By "git log" check out if you are deploying correct versions, do it for submodules too. The do "scap deploy -v". Log it in #wikimedia-operations.

Latest revision as of 07:40, 2 August 2016

Redirect to: