You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Toolsadmin.wikimedia.org/Build
The Striker build and deploy process involves updating the labs-striker-deploy git repository and then pushing the repository out to servers running Striker via scap3.
Deploy directory structure
/srv/deployment/striker/deploy/ ├── _build/ # Generated by make_wheels.sh, only on Cloud VPS staging server | └── venv/ # virtualenv used for creating wheels & top level ├── ddl/ # Manually created SQL patches ├── make_wheels.sh # Build script for preparing labs-striker-wheels patches ├── public_html/ # Apache/nginx docroot | └── static/ # labs-striker-staticfiles submodule ├── requirements.txt # `pip --freeze` generated requirements ├── scap/ | ├── checks/ | | └── virtualenv.sh # Check script to rebuild venv on target hosts | ├── checks.yaml # Post-deploy checks for Scap3 | ├── dsh/ # Scap3 target host sets | ├── log/ # Generated by scap3, only on deploy master | └── scap.cfg # Scap3 config file ├── striker/ # striker submodule | └── requirements.txt # Product level requirements └── wheels/ # labs-striker-wheels submodule
Submodules
Most of the interesting code changes will happen in submodules of labs-striker-deploy:
- labs-striker: The Striker Django application
- labs-striker-staticfiles: Django managed static files (css, js, images)
- labs-striker-wheels: Python binary packages in wheel format
labs-striker
There is nothing special to do to prepare this repo for deployment. Merge commits using Gerrit.
Changes made here however will trigger changes to other submodules. The the point a release is being made:
- labs-striker-staticfiles must be updated if any changes were made to requirements.txt or if files were modified in the
static
directory. - labs-striker-wheels must be updated if any changes were made to requirements.txt.
labs-striker-staticfiles
These static assets are collected from the installed Django applications using Django's collectstatic
management script. The contrib/collectstatic.sh
convenience script is provided in labs-striker for collecting the files. This can be run in a MediaWiki-Vagrant development environment.
$ cd /vagrant/srv/striker
$ contrib/collectstatic.sh
Deleting 'robots.f71d20196d4c.txt'
Deleting 'robots.txt'
Deleting 'staticfiles.json'
...
Post-processed 'robots.txt' as 'robots.f71d20196d4c.txt'
110 static files copied to '/vagrant/srv/striker/staticfiles', 110 post-processed.
$ cd staticfiles
$ python -mjson.tool staticfiles.json > staticfiles.json.pretty
$ mv staticfiles.json.pretty staticfiles.json
$ cp -R . $PATH_TO_LAB_STRIKER_STATICFILES
$ cd $PATH_TO_LAB_STRIKER_STATICFILES
$ git status
$ git add .
$ git commit
$ git review
Before submitting you may want to look for old assets that can be removed as well. These would generally be <FILENAME>.<CONTENT_HASH>.<EXT>
files that are for versions of the files that were removed from the manifest in a prior update that has already been deployed.
Testing in Cloud VPS project
TODO: explain how I typically stage pending changes in the striker Cloud VPS project.
Deploy
This section describes the process of getting the live service modified with new changes.
- download the source code git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker
- download the deployment git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker/deploy
- develop: make your code changes, tests, etc (out of the scope of this section)
- git review the source code (gerrit) and get it merged
- update the submodule commit in the deployment repo pointing to the commit you want to deploy
- https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker is a submodule in the deploy repo
- Occasionally this will not fetch no matter what you do, so I found that running
git submodule update --recursive --init
captured the latest code, allowing one to cd into the striker subdirectory and rungit checkout <desired commit>
which seems to be how this is often updated. - Add, commit and submit your changes for review on the deploy repo. This is the repo you are deploying!
- ssh jump to deployment server, for example deploy1001.eqiad.wmnet and cd to /srv/deployment/stricker/deploy
- git fetch & merge/rebase & git submodule update --recursive
- The submodule won't update unless you changed it on the deploy repository in the last step above
scap deploy