You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Toolsadmin.wikimedia.org/Build: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>BryanDavis
(Add an overview of the deploy directory structure)
imported>Bstorm
Line 67: Line 67:


== Deploy ==
== Deploy ==
TODO: explain basic scap3 deployment
 
This section describes the process of getting the live service modified with new changes.
 
* download the source code git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker
* download the deployment git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker/deploy
* develop: make your code changes, tests, etc (out of the scope of this section)
* git review the source code (gerrit) and get it merged
* update the submodule commit in the deployment repo pointing to the commit you want to deploy
** https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker is a submodule in the deploy repo
** Occasionally this will not fetch no matter what you do, so I found that running <pre>git submodule update --recursive --init</pre> captured the latest code, allowing one to cd into the striker subdirectory and run <pre>git checkout <desired commit></pre> which seems to be how this is often updated.
** Add, commit and submit your changes for review on the deploy repo.  This is the repo you are deploying!
* ssh jump to deployment server, for example '''deploy1001.eqiad.wmnet''' and cd to '''/srv/deployment/stricker/deploy'''
* git fetch & merge/rebase & git submodule update --recursive
** The submodule won't update unless you changed it on the deploy repository in the last step above
* <pre>scap deploy</pre>

Revision as of 15:31, 14 August 2018

The Striker build and deploy process involves updating the labs-striker-deploy git repository and then pushing the repository out to servers running Striker via scap3.

Deploy directory structure

/srv/deployment/striker/deploy/
 ├── _build/                    # Generated by make_wheels.sh, only on Cloud VPS staging server
 |    └── venv/                 # virtualenv used for creating wheels & top level 
 ├── ddl/                       # Manually created SQL patches
 ├── make_wheels.sh             # Build script for preparing labs-striker-wheels patches
 ├── public_html/               # Apache/nginx docroot
 |    └── static/               # labs-striker-staticfiles submodule
 ├── requirements.txt           # `pip --freeze` generated requirements
 ├── scap/
 |    ├── checks/
 |    |    └──  virtualenv.sh   # Check script to rebuild venv on target hosts
 |    ├── checks.yaml           # Post-deploy checks for Scap3
 |    ├── dsh/                  # Scap3 target host sets
 |    ├── log/                  # Generated by scap3, only on deploy master
 |    └── scap.cfg              # Scap3 config file
 ├── striker/                   # striker submodule
 |    └── requirements.txt      # Product level requirements
 └── wheels/                    # labs-striker-wheels submodule

Submodules

Most of the interesting code changes will happen in submodules of labs-striker-deploy:

labs-striker

There is nothing special to do to prepare this repo for deployment. Merge commits using Gerrit.

Changes made here however will trigger changes to other submodules. The the point a release is being made:

  • labs-striker-staticfiles must be updated if any changes were made to requirements.txt or if files were modified in the static directory.
  • labs-striker-wheels must be updated if any changes were made to requirements.txt.

labs-striker-staticfiles

These static assets are collected from the installed Django applications using Django's collectstatic management script. The contrib/collectstatic.sh convenience script is provided in labs-striker for collecting the files. This can be run in a MediaWiki-Vagrant development environment.

$ cd /vagrant/srv/striker
$ contrib/collectstatic.sh
Deleting 'robots.f71d20196d4c.txt'
Deleting 'robots.txt'
Deleting 'staticfiles.json'
...
Post-processed 'robots.txt' as 'robots.f71d20196d4c.txt'

110 static files copied to '/vagrant/srv/striker/staticfiles', 110 post-processed.
$ cd staticfiles
$ python -mjson.tool staticfiles.json > staticfiles.json.pretty
$ mv staticfiles.json.pretty staticfiles.json
$ cp -R . $PATH_TO_LAB_STRIKER_STATICFILES
$ cd $PATH_TO_LAB_STRIKER_STATICFILES
$ git status
$ git add .
$ git commit
$ git review

Before submitting you may want to look for old assets that can be removed as well. These would generally be <FILENAME>.<CONTENT_HASH>.<EXT> files that are for versions of the files that were removed from the manifest in a prior update that has already been deployed.

Testing in Cloud VPS project

TODO: explain how I typically stage pending changes in the striker Cloud VPS project.

Deploy

This section describes the process of getting the live service modified with new changes.

  • download the source code git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker
  • download the deployment git repository: https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker/deploy
  • develop: make your code changes, tests, etc (out of the scope of this section)
  • git review the source code (gerrit) and get it merged
  • update the submodule commit in the deployment repo pointing to the commit you want to deploy
    • https://gerrit.wikimedia.org/r/#/admin/projects/labs/striker is a submodule in the deploy repo
    • Occasionally this will not fetch no matter what you do, so I found that running
      git submodule update --recursive --init
      captured the latest code, allowing one to cd into the striker subdirectory and run
      git checkout <desired commit>
      which seems to be how this is often updated.
    • Add, commit and submit your changes for review on the deploy repo. This is the repo you are deploying!
  • ssh jump to deployment server, for example deploy1001.eqiad.wmnet and cd to /srv/deployment/stricker/deploy
  • git fetch & merge/rebase & git submodule update --recursive
    • The submodule won't update unless you changed it on the deploy repository in the last step above
  • scap deploy