You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Gerrit/Upgrade

From Wikitech-static
< Gerrit
Revision as of 20:04, 10 May 2021 by imported>Hashar ({{gerrit nav}})
Jump to navigation Jump to search

Since Feb 2nd 2021, we deploy the upstream war file as is. It is fetched from https://www.gerritcodereview.com/ . We still build extra plugins from sources though using the source code held in the Gerrit repository operations/software/gerrit.

The upstream branches (master, stable-3.2) are manually kept in sync. We have a forked branch wmf/stable-3.2 which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository.

Build prerequisites

Install OpenJDK 8, git, NodeJS, Python 3. On Debian:

$ sudo apt install git openjdk-8-jdk-headless nodejs python3

Gerrit and its plugins are build using Bazel. Since the exact version to use might differ between branches or can be updated by upstream at any point, one should install Bazelisk. It is a frontend to Bazel which automatically installs the needed Bazel version for the Gerrit branch you are working on. You will require the Go programming language in order to build it. The following will build Bazelisk 1.6.1 from source and copy the resulting binary in your user bin directory (~/.local/bin):

$ which go || sudo apt install golang-go
$ mkdir /tmp/bazelisk
$ (/tmp/bazelisk-build && go mod init . && go get github.com/bazelbuild/bazelisk@v1.6.1)
$ mkdir -p $HOME/.local/bin && cp /tmp/bazelisk-build/gopath/bin/bazelisk $HOME/.local/bin/bazelisk
$ rm -fR /tmp/bazelisk-build

Update our repository

Upgrade Gerrit core branch

  1. Clone Gerrit and submodules from Wikimedia add upstream as a remote
    git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
    cd gerrit
    git remote add upstream https://gerrit.googlesource.com/gerrit
    
    Our repository HEAD points to the branch holding the material for the deployment deploy/wmf/stable-3.2
  2. Checkout the stable-3.2 branch (as of November 2020) which is a copy of the upstream release we currently use:
    git checkout stable-3.2 origin/stable-3.2
    
  3. Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the stable-3.2 directly to Gerrit:
    git fetch --tags upstream stable-3.2
    git merge --ff-only v3.2.3
    git push --tags origin HEAD:stable-3.2
    
  4. Checkout wmf/stable-3.2 and merge the latest tag, push the resulting SINGLE merge commit up for review:
    git checkout wmf/stable-3.2
    git merge v3.2.8
    

Update bundled submodules

Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The .gitmodules file comes from upstream and uses relative urls. Since your work copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting url.<base>.insteadOf = <our url>.

  1. Ensure you are in our forked branch:
    git checkout wmf/stable-3.2
    # update as needed
    
  2. Update git submodules from the upstream repository:
    git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
    
  3. Push for review. At this point, you will likely be ahead of wmf/stable-3.2 by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:
    git commit --ammend  # to add Change-Id and Phabricator tracking bug
    git push origin HEAD:refs/for/wmf/stable-3.2
    

Fetch our additional plugins

As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered and they need to be updated to match the new Gerrit core. You will need ssh admin access to Gerrit to be able to list the installed plugins as well as jq (sudo apt install jq).

List our plugins:

$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$


Most plugins come with a set of stable- branches, some support multiple versions and solely have a master branch. The proper branch to use for our extra plugins is indicated in .gitmodules , we thus just have to update them from the remote branch (git submodule update --remote plugins/foo). To update our plugins:

$ ./wmf-plugins-update.sh
Updating Wikimedia specific plugins
<the list of our plugins is shown here>
<git submodule update if relevant>
Done
$

Fix-up of gitiles plugin

TODO we should fork it.

The gitiles plugin per default fetches fonts directly from Google. Since that's a third party dependency that we do not want, we instead want gitiles to use the fonts that Gerrit itself offers.

To achieve that, make sure your gitiles code includes change I4f4a0b7dd575cbc27641063e05d13b8a43a51d8b. The change did not get much traction upstream, so it probably won't land there. We nonetheless want to include that change.

Fix-up of javamelody plugin

TODO should not be needed

The javamelody plugin comes with external dependencies. Gerrit's build system does not automatically pick them up. To allow Gerrit to build the plugin, merge the contents of plugins/javamelody/external_plugin_deps.bzl into Gerrit's own plugins/external_plugin_deps.bzl


Once done updating our plugins submodules, git add them, update your commit and send it for review:

git add $(./wmf-plugins-list.sh)
git commit --amend
git push origin HEAD/refs/for/wmf/stable-3.2

Build Gerrit

Gerrit core

As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use upstream .war.

  1. Checkout the wmf/stable-3.2 (as of November 2020)
    git checkout -b wmf/stable-3.2 origin/wmf/stable-3.2
    

Production uses Java 8 as of November 19th 2020 which we will eventually switch to Java 11 ( https://phabricator.wikimedia.org/T268225 ). Meantime, Bazel has to be pointed to your local Java 8 installation: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64.

  1. Build the bazel-bin/release.war
    JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bazel build release
    

Note that if your building Gerrit v3.3 (or later) or master from 2020-10-06 (or later) and you're targeting Java 8, you'll also have to add the argument --java_toolchain=//tools:error_prone_warnings_toolchain (this is untested, but claimed on the gerrit mailing list).

Plugins

Most plugins come with build instructions usually available at src/main/resources/Documentation/build.html.

Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. The wmf-plugins-build.sh script iterate over the plugins found via wmf-plugins-list.sh and build them, taking care of dependencies files if needed:

$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$

Build them:

$ ./wmf-plugins-build.sh
Building Wikimedia specific plugins

> Building go-import

...

Done building
Copying build artifacts
plugins-wmf/go-import.jar
plugins-wmf/healthcheck.jar
plugins-wmf/its-base.jar
plugins-wmf/its-phabricator.jar
plugins-wmf/javamelody.jar
plugins-wmf/lfs.jar
plugins-wmf/metrics-reporter-jmx.jar
plugins-wmf/metrics-reporter-prometheus.jar
plugins-wmf/motd.jar
plugins-wmf/reviewers.jar
plugins-wmf/zuul.jar

Those should be copied to the wmf/deploy branch

$

Upload artifacts

  1. Checkout the deploy/wmf/stable-X branch that aligns with the build branch
  2. Copy all the plugin jars from ./plugins-wmfover to ./plugins in the deploy branch
  3. Ensure Gerrit bundled plugins are not included and that any plugins we no more have installed are removed
  4. Commit the changes and upload to Gerrit
  5. Run ./upload_artifacts.py --version=3.2.8 gerrit.war plugins/foo.jar ... for the core application and all updated plugins (cf: Archiva, specifically setting up your ~/.m2/settings.xml
    Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.

Deploying

  1. SSH to the deployment master (ssh deployment.eqiad.wmnet)
  2. Navigate to /srv/deployment/gerrit/gerrit
  3. Fetch & checkout the appropriate deploy/wmf/stable-X branch
  4. Deploy to gerrit2001 to ensure that there are no errors with git-fat et al scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'
  5. Deploy to gerrit1001 scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'
  6. If you're only deploying plugins, you're done, otherwise SSH to the Gerrit master and issue sudo service gerrit restart

XXX ensure plugins bundled in the war files get installed: java -jar gerrit.war init --install-all-plugins

XXX add a warning that init would cause upgrade steps to be conducted if need be (aka from a Gerrit version to another)