You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Gerrit/Upgrade

From Wikitech-static
< Gerrit
Revision as of 20:40, 29 April 2021 by imported>Hashar (Created page with " Since Feb 2nd 2021, we use the upstream war file fetched from https://www.gerritcodereview.com/ but we still build plugins from sources. The source code is held in the Gerrit...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Since Feb 2nd 2021, we use the upstream war file fetched from https://www.gerritcodereview.com/ but we still build plugins from sources. The source code is held in the Gerrit repository operations/software/gerrit.

Prerequisites

  1. Install OpenJDK 8 and git (openjdk-8-jdk and git packages on Debian)
  2. Bazel is required (as of July 2020 at least bazel 3.4.1, can be checked in.bazelversion ).

Instead of installing Bazel directly, consider installing Bazelisk, which is a Bazel frontend that automatically installs the needed Bazel version for the Gerrit branch you are working on. Bazelisk saves you a lot of trouble when building Gerrit, so if you can, use it.

Update our repository

  1. Clone gerrit and submodules and add upstream as a remote
    git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
    cd gerrit
    git remote add upstream https://gerrit.googlesource.com/gerrit
    
    Our repository HEAD points to the branch holding the material for the deployment deploy/wmf/stable-3.2
  2. Checkout the stable-3.2 branch (as of November 2020) that tracks the current upstream release:
    git checkout stable-3.2 origin/stable-3.2
    
  3. Fetch the upstream branch and fast-forward our copy of their branch to the latest tag and push the stable-3.2 directly to Gerrit:
    git fetch --tags upstream stable-3.2
    git merge --ff-only v3.2.3
    git push --tags origin HEAD:stable-3.2
    
  4. Checkout wmf/stable-3.2 and merge the latest tag, push the resulting SINGLE merge commit up for review:
    git checkout wmf/stable-3.2
    git merge v3.2.8
    

Next update git submodules. The .gitmodules file comes from upstream and uses relative urls. Since your work copy points to our Gerrit, the submodules would be fetched from our Gerrit. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git setting url.<base>.insteadOf = <our url>:

  1. Update git submodules from the upstream repository:
    git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
    
  2. Push for review. At this point, you will likely be ahead of wmf/stable-3.2 by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:
    git commit --ammend  # to add Change-Id and Phabricator tracking bug
    git push origin HEAD:refs/for/wmf/stable-3.2
    

Fetch our additional plugins

As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we need to fetch them. Some plugins support out-of-tree builds, but the most stable approach is to simply clone them into directories underneath Gerrit's plugins directory in a checkout of our wmf/stable-3.2 branch. You can use the following short bash script (which assumes you can ssh to our Gerrit and you have got jq installed) to clone the needed repositories:

#!/bin/bash

set -e
set -o pipefail

INSTALLED_PLUGINS="$(ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls --format JSON | jq --raw-output 'keys | .[]')"
DEPENDENCIES_FOR_BUILDING="its-base"

cd plugins

for PLUGIN in $INSTALLED_PLUGINS $DEPENDENCIES_FOR_BUILDING
do
  if [ ! -e "$PLUGIN" ]
  then
    echo "Cloning $PLUGIN ..."
    git clone "https://gerrit.googlesource.com/plugins/$PLUGIN"
  fi
done

echo "Success. All needed plugins cloned."


Knowing which branch to choose for which repository is not so straight forward. If a plugin (most don't) provide version tags, use them if you are building Gerrit for a specific version. Otherwise, many plugins come with stable- branches. Use the one for the Gerrit version you are building. If a plugin does not offer any of these, try the master branch.

Please go through each of the plugins individually to find the tag or branch that best fits the desired target. There are only 22 or so plugins, so while it is annoying, it will not take ages.

The following script go through each of our installed plugins, attempts to checkout the tag, else the stable branch and finally fallback to master. On completion it list for each plugin the git description (git describe --all) and the checked out branch.

#!/bin/bash

set -eu -o pipefail

if [ ! $# -eq 1 ]; then
  echo "Usage: $0 <gerrit git tag>"
  exit 2
fi

TAG=$1
INSTALLED_PLUGINS="$(ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls --format JSON | jq --raw-output 'keys | .[]')"

echo "Checkout tag or branch or master..."
for PLUGIN in $INSTALLED_PLUGINS; do
  (
    cd "plugins/$PLUGIN"
    git checkout "$TAG" || git checkout origin/stable-3.2 || git checkout origin/master
  );
done;
echo "Done."

echo "Show the resolved branch or tag if HEAD is detached"
for PLUGIN in $INSTALLED_PLUGINS; do
  (
    cd "plugins/$PLUGIN"
    echo "-- $PLUGIN --"
    git describe --all
    git branch --points-at=HEAD
    )
done;
echo "Done."

Fix-up Gerrit core submodules plugin

The plugins listed as submodules might be ahead of the tag/branch available and the heuristic above might have rolled them back in the past., for example when a plugin did not receive a tag. In this case, you should reinitialize the submodules:

git submodule update

Fix-up of gitiles plugin

The gitiles plugin per default fetches fonts directly from Google. Since that's a third party dependency that we do not want, we instead want gitiles to use the fonts that Gerrit itself offers.

To achieve that, make sure your gitiles code includes change I4f4a0b7dd575cbc27641063e05d13b8a43a51d8b. The change did not get much traction upstream, so it probably won't land there. We nonetheless want to include that change.

Fix-up of javamelody plugin

The javamelody plugin comes with external dependencies. Gerrit's build system does not automatically pick them up. To allow Gerrit to build the plugin, merge the contents of plugins/javamelody/external_plugin_deps.bzl into Gerrit's own plugins/external_plugin_deps.bzl


Most plugins come with build instructions usually available at src/main/resources/Documentation/build.html.

Build Gerrit

Gerrit core

As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use upstream .war.

  1. Checkout the wmf/stable-3.2 (as of November 2020)
    git checkout -b wmf/stable-3.2 origin/wmf/stable-3.2
    

Production uses Java 8 as of November 19th 2020 which we will eventually switch to Java 11 ( https://phabricator.wikimedia.org/T268225 ). Meantime, Bazel has to be pointed to your local Java 8 installation: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64.

  1. Build the bazel-bin/release.war
    JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bazel build release
    

Note that if your building Gerrit v3.3 (or later) or master from 2020-10-06 (or later) and you're targeting Java 8, you'll also have to add the argument --java_toolchain=//tools:error_prone_warnings_toolchain (this is untested, but claimed on the gerrit mailing list).

Plugins

Since we use the upstream jar, we do not need to build the plugins listed as git submodules. The following script retrieve the list of our installed plugins, extract the bundled plugins from .gitmodules and remove them from the list of plugins to build.

#!/bin/bash

set -eu
set -o pipefail

WAR_BUNDLED_PLUGINS=$(git config -f .gitmodules --get-regexp 'submodule\.plugins/.*\.path'|cut -d\  -f2|cut -d/ -f2)
INSTALLED_PLUGINS="$(ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls --format JSON | jq --raw-output 'keys | .[]')"

TO_BUILD=$(/usr/bin/comm -13 <(echo "$WAR_BUNDLED_PLUGINS") <(echo "$INSTALLED_PLUGINS"))
printf "== List of our extra plugins ==\n%s\n" "$TO_BUILD"


for PLUGIN in $TO_BUILD
do
  printf "=== Building %s ===\n" "$PLUGIN"
  JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \
    bazel build --workspace_status_command="python ./tools/workspace_status_release.py" plugins/"$PLUGIN"
done;

Troubleshooting

Current Bazel version is 0.18.1, expected at least 0.19.0
Bazel updates all the time, update your bazel version from the bazel website (ensure you verify the version signature with the [ https://bazel.build/bazel-release.pub.gpg bazel-release.pub.gpg] key. Or even better: Use Bazelisk, which is a transparent wrapper for Bazel that takes care of preparing the correct version for you.
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
bazel shells out to curl to download things and, sadly, repo.eclipse.org only supports TLSv1 openssl s_client -connect repo.eclipse.org:443 -tls1. You can modify your openssl settings in /etc/ssl/openssl.cnf. If your MinProtocol is set to TLSv1.2 (because you're awesome) — change it to TLSv1 (because not everyone is awesome). Be sure to change it back when you're done.

Upload artifacts

  1. Copy the resulting bazel-bin/release.war and bazel-bin/plugins/$PLUGIN/$PLUGIN.jar to somewhere safe
  2. Checkout the deploy/wmf/stable-X branch that aligns with the build branch
  3. Copy in the release.war you saved as ./gerrit.war
  4. Copy all the plugin jars over to ./plugins in the deploy branch
  5. Commit the changes, upload to Gerrit, etc
  6. Run ./upload_artifacts.py --version=2.15.2 gerrit.war plugins/foo.jar ... for the core application and all updated plugins (cf: Archiva, specifically setting up your ~/.m2/settings.xml
    Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins so I wrote the tool above

Deploying

  1. SSH to the deployment master
  2. Navigate to /srv/deployment/gerrit/gerrit
  3. Fetch & checkout the appropriate deploy/wmf/stable-X branch
  4. Deploy to gerrit2001 to ensure that there are no errors with git-fat et al scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'
  5. Deploy to gerrit1001 scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'
  6. If you're only deploying plugins, you're done, otherwise SSH to the Gerrit master and issue sudo service gerrit restart