You are browsing a read-only backup copy of Wikitech. The live site can be found at

Fundraising/Cluster/Deployments: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
(moved from mediawiki)
No edit summary
Line 199: Line 199:
== CentralNotice ==
== CentralNotice ==
Please see [[wikitech:CentralNotice|CentralNotice documentation on Wikitech]].
Please see [[wikitech:CentralNotice|CentralNotice documentation on Wikitech]].
== Big Civicrm Upgrades ==

Revision as of 00:56, 19 January 2022

Release tags and branches

Please note that many repositories include checked-in external libraries. This should only be done on the deployment branches, the master branch is rebuilt by the developer. Do check in composer.lock and similar files for version locking.

Repository Deployment branch
crm deployment
mediawiki/core fundraising/REL1_31 (or current LTS)
DonationInterface deployment
CentralNotice wmf_deploy
SmashPig deployment
tools deploy
dash deployment
all others master or as submodules

Deployment process

Fundraising cluster deployment is always done from frpm1001. Software is deployed using the custom fundraising_code_update and rsync_blaster scripts, configuration is deployed with frpm1001 as the puppetmaster. More information is available about deployment on Collab.

Our deployment scripts simply pull down specific branches from gerrit and rsync them to our production servers. We never run composer nor any minification scripts in production. This means that we need to check in all our dependencies to our repositories, generally in a /vendor submodule under the repo with the dependencies.

For payments-wiki, we deploy the fundraising/REL1_35 branch (updated every two years to track the LTS branch of Mediawiki) of mediawiki/core, as well as the fundraising/REL1_35 branch of mediawiki/core/vendor. Our branch of mediawiki/core/vendor includes dependencies for Mediawiki core and our deployed extensions but no dev-dependencies. We do not deploy additional vendor subdirectories under each extension, as we make use of the composer-merge plugin to install all dependencies to the top-level vendor directory.

We generally update mediawiki/core/vendor one library at a time. For example, if we have tagged an updated version of SmashPig, the procedure to update vendor is as follows:

  # From the root mediawiki directory:
  composer update --no-dev wikimedia/smash-pig
  cd vendor
  git add .
  git commit -m "Update SmashPig"
  # send for review and merge the vendor commit in gerrit
  cd ..  # back to the mediawiki root directory
  git add vendor composer.lock
  git commit -m "Update SmashPig library and vendor directory"


Change paymentswiki configuration files on the frack puppetmaster, frpm1001. Check out the settings repo (into your homedir on frpm, not your dev machine).

 developer@frpm1001:~$ git clone /var/lib/git/localsettings.git

Then inside the copy in your homedir, make changes and push to the shared repo. After pushing to the shared repo, you can ask another fr-tech member to review the change if there's anything tricksy. Then deploy the change using fundraising_code_update and rsync_blaster, targeting whichever project pertains to the config file you have just changed.

fr-tech doesn't have access to configure some subsystems directly - any settings outside of the /srv directory are managed by fr-tech-ops via puppet. You can clone the puppet repo down to your frpm homedir as well but you will have to ask an fr-tech-ops member to actually apply any changes.

Matching gifts employers list

The csv containing employer names and their Civi IDs is deployed alongside the LocalSettings.php configuration file for payments-wiki. You need to copy it from the civi box to the frpm box.

On frpm you can copy a script to update it from the local scripts repo in /var/lib/git.

 git clone /var/lib/git/tools.git/ bin

When you run this script, you will need to be ssh'ed into the puppetmaster with agent forwarding (ssh -A frpm) so that your user on frpm can access files on the civi box. Run ~/bin/ from your local copy of the localsettings git repo. It will

  • copy the file from where it is generated on Civi to payments-wiki/employers.csv
  • commit it to git
  • run fundraising_code_update and rsync_blaster for payments-wiki


There are three ways to deploy SmashPig: standalone, in CiviCRM, and in Payments-wiki.

Deploying Standalone

Deploy the standalone 'SmashPig' project when you need to make code changes to the IPN listener or Pending queue consumer, or when you want to change SmashPig-level config settings on all machines on the cluster.

  • For code, merge the changes you want to deploy from the master branch to the deployment branch
    • If you have changed composer.lock on master, update the vendor submodule with updated dependencies and update the pointer on development. You can update vendor in its own commit or just amend the merge commit to include the vendor update. Be sure not to check in dev-dependencies.
  • For settings changes, make your changes in the SmashPig/local-config subdirectory of your local checkout of localsettings, commit, and push, as with any other config changes.
  • On frpm1001, use fundraising_code_update and rsync_blaster with the "SmashPig" project as a target

Updating the Composer Package

Under CiviCRM and DonationInterface, SmashPig is included as a composer package.

  • We distribute SmashPig as a composer package wikimedia/smash-pig, managed at Template:Packagist
  • Version numbers are controlled by git tags using the semantic versioning numbering scheme (breaking changes increase the 1st digit, minor changes increase the second digit, bugfixes increase the 3rd digit).
  • To bump the package version from e.g. 0.5.9 to 0.5.10:

1. Checkout master in your local copy of the SmashPig repo

2. Update the git tag

    git tag v0.5.10 HEAD
    git push origin v0.5.10

3. Go to Template:Packagist

4. Click the green 'Update' button

Deploying in CiviCRM

Updating composer dependencies in the crm repo is a bit involved. After updating the composer package as described above,

  • in crm/master, composer update wikimedia/smash-pig
    • if composer doesn't see the new version, try composer clear-cache
  • commit composer.lock change to crm/master
  • check out crm/deployment
  • merge crm/master
  • rm -rf vendor # to get rid of dev dependencies
  • git submodule update --init vendor # to get a copy of vendor identical to the repo
  • composer install --no-dev
  • cd vendor
  • git add . && git commit
  • git review and merge to master in the vendor repo
  • update submodule ref for vendor in crm/deployment
  • commit crm/deployment (git commit --amend to squash it in with the merge commit)

Example patches:

Deploying in Payments-wiki

This differs from updating in CRM, in that

  • we do not specify the smash-pig requirement in the root mediawiki composer.json but rather in extensions/DonationInterface/composer.json. The composer merge plugin pulls in requirements from the extensions and installs them to the mediawiki/vendor directory.
  • We only maintain one branch of mediawiki for payments, fundraising/REL1_35 (or whichever RELX_YY represents the latest LTS). So for most updates there is no 'merge to deployment' step

If you are drastically changing the version requirement (e.g. from 1.x to 2.x) you will need to change that in composer.json in the DonationInterface repo, merge that change to DI's deployment branch, and update the DonationInterface submodule pointer as well as all the steps below. If you only want to deploy a minor update, you can just update the composer.lock file and vendor submodule in the top-level payments-wiki source checkout.

  1. Ensure your vendor directory is consistent with the latest changes on mediawiki/vendor's fundraising/REL1_35 branch
    1. Delete the vendor directory from the root of your payments-wiki source (the mediawiki repo at branch fundraising/REL1_35)
    2. Pull down the pristine copy
      git submodule update --init vendor
  2. Update smash-pig using composer
    composer update --no-dev wikimedia/smash-pig
  3. Go into /vendor and
    git add .
    git commit
    git review
  4. Make sure the vendor commit is merged in gerrit, then back in the root of your payments-wiki source:
    git add vendor
    git add composer.lock
    git commit
    git review
  5. Once that commit is merged in gerrit, deploy the payments-wiki project from frpm1001


Work In Progress, feel free to edit/add/rewrite

Hint: you can alway mock other's most recent deploy process from Gerrit to see if everything looks smooth, and if jenkis did not catch your commit, just send a comment "recheck".

. Workflow: submit patches to master branch, when deploying they go to deployment branch. Deployment branches don't have unit tests.

Part 1:

1. Make sure your master and deployment branches are up to date

2. Go onto the deployment branch (if you do not have development branch yet, just do

git checkout -b deployment --track origin/deployment) and find the differences between deployment and master. The below command will output them in plain text:

git log --oneline --no-merges --reverse deployment..master | cat

3. Copy the difference in commits between deployment and master as it would be used in another step.

4. Merge master into deployment git merge master

5. There may be merge conflicts in tests, they need to be removed from the deployment branch. This could be done using the git mergetool command

6. On the git mergetool interface, ensure the "(d)eleted file" option is selected for test files.

7. When finished, enter git commit

8. This brings you into a commit message, paste in the differences from step 2 underneath the Merge message

9. Push up the changes to gerrit git review

10. Once they are on gerrit, you can self +2 it if you feel comfortable with the code changes made. You can always ask others if you aren't sure what one of the patches is.

Part 2: Submodule Update

This is needed as we deploy the entire payments project instead of just Donation Interface when changes are made.

1. In the main payments wiki folder, make sure you have the latest code at the right branch (you can always find the right branch from other's most recent deploy task). If you are running into issues and part 1 has been merged, you can do a git reset --hard

2. Add the updated Donation Interface extension git add extensions/DonationInterface This will change the pointer of the submodule.

3. git commit The commit message can be 'Update Donation Interface submodule'

4. Push up the changes to gerrit git review

5. Once they are on gerrit, you can self +2


Please see CentralNotice documentation on Wikitech.

Big Civicrm Upgrades