Toolsadmin.wikimedia.org/Build

The Striker build and deploy process involves updating the labs-striker-deploy git repository and then pushing the repository out to servers running Striker via scap3.

Deploy directory structure

/srv/deployment/striker/deploy/
 ├── _build/                    # Generated by make_wheels.sh, only on Cloud VPS staging server
 |    └── venv/                 # virtualenv used for creating wheels & top level 
 ├── ddl/                       # Manually created SQL patches
 ├── make_wheels.sh             # Build script for preparing labs-striker-wheels patches
 ├── public_html/               # Apache/nginx docroot
 |    └── static/               # labs-striker-staticfiles submodule
 ├── requirements.txt           # `pip --freeze` generated requirements
 ├── scap/
 |    ├── checks/
 |    |    └──  virtualenv.sh   # Check script to rebuild venv on target hosts
 |    ├── checks.yaml           # Post-deploy checks for Scap3
 |    ├── dsh/                  # Scap3 target host sets
 |    ├── log/                  # Generated by scap3, only on deploy master
 |    └── scap.cfg              # Scap3 config file
 ├── striker/                   # striker submodule
 |    └── requirements.txt      # Product level requirements
 └── wheels/                    # labs-striker-wheels submodule

Submodules

Most of the interesting code changes will happen in submodules of labs-striker-deploy:

• labs-striker: The Striker Django application
• labs-striker-staticfiles: Django managed static files (css, js, images)
• labs-striker-wheels: Python binary packages in wheel format

labs-striker

There is nothing special to do to prepare this repo for deployment. Merge commits using Gerrit.

Changes made here however will trigger changes to other submodules. The the point a release is being made:

• labs-striker-staticfiles must be updated if any changes were made to requirements.txt or if files were modified in the static directory.
• labs-striker-wheels must be updated if any changes were made to requirements.txt.

labs-striker-staticfiles

These static assets are collected from the installed Django applications using Django's collectstatic management script. The contrib/collectstatic.sh convenience script is provided in labs-striker for collecting the files. This can be run in a MediaWiki-Vagrant development environment.

$ cd /vagrant/srv/striker
$ contrib/collectstatic.sh
Deleting 'robots.f71d20196d4c.txt'
Deleting 'robots.txt'
Deleting 'staticfiles.json'
...
Post-processed 'robots.txt' as 'robots.f71d20196d4c.txt'

110 static files copied to '/vagrant/srv/striker/staticfiles', 110 post-processed.
$ cd staticfiles
$ python -mjson.tool staticfiles.json > staticfiles.json.pretty
$ mv staticfiles.json.pretty staticfiles.json
$ cp -R . $PATH_TO_LAB_STRIKER_STATICFILES
$ cd $PATH_TO_LAB_STRIKER_STATICFILES
$ git status
$ git add .
$ git commit
$ git review

Before submitting you may want to look for old assets that can be removed as well. These would generally be <FILENAME>.<CONTENT_HASH>.<EXT> files that are for versions of the files that were removed from the manifest in a prior update that has already been deployed.

Testing in Cloud VPS project

TODO: explain how I typically stage pending changes in the striker Cloud VPS project.

Deploy

This section describes the process of getting the live service modified with new changes.

• download the source code git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker
• download the deployment git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker/deploy
• develop: make your code changes, tests, etc (out of the scope of this section)
• git review the source code (gerrit) and get it merged
• update the submodule commit in the deployment repo pointing to the commit you want to deploy
  • https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker is a submodule in the deploy repo
  • Occasionally this will not fetch no matter what you do, so I found that running

    git submodule update --recursive --init

    captured the latest code, allowing one to cd into the striker subdirectory and run

    git checkout <desired commit>

    which seems to be how this is often updated.
  • Add, commit and submit your changes for review on the deploy repo. This is the repo you are deploying!
• ssh jump to deployment server, for example deploy1001.eqiad.wmnet and cd to /srv/deployment/stricker/deploy
• git fetch & merge/rebase & git submodule update --recursive
  • The submodule won't update unless you changed it on the deploy repository in the last step above

• scap deploy