View source for Gerrit/Upgrade

{{gerrit nav}}
Since Feb 2nd 2021, we deploy the upstream war file as is. It is fetched from https://www.gerritcodereview.com/ , stored using git-fat in {{git repo|operations/software/gerrit}} under a <code>deploy/wmf/stable-X.Y</code> branch. We still build extra plugins from sources, the build relies on the source code held in the Gerrit repository {{git repo|operations/software/gerrit}} in branch <code>wmf/stable-X.Y</code>

The upstream branches (<code>master</code>, <code>stable-3.2</code>) are manually kept in sync. We have a forked branch <code>wmf/stable-3.2</code> which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository. Our branch can be used to cherry pick patches for plugins build purpose, but must eventually be send to upstream, reviewed and merged there. The branch overview:

{| class="wikitable"
|+ operations/software/gerrit branches layout
! Branch !! Description
|-
| <code>stable-3.2</code> || Gerrit core, updated from upstream
|-
| <code>wmf/stable-3.2</code> || Fork holding our build script and our plugins as submodules
|-
| <code>wmf/deploy/stable-3.2</code> || scap deployment repository holding Gerrit .war and plugins .jar with git-fat
|}

== Build prerequisites ==

''The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.2/Documentation/dev-bazel.txt''

Install OpenJDK 8, git, NodeJS, Python 3. On Debian:
<syntaxhighlight lang="console">
$ sudo apt install git openjdk-8-jdk-headless nodejs python3
</syntaxhighlight>Gerrit and its plugins are build using [https://docs.bazel.build/versions/master/install.html Bazel]. Since the exact version to use might differ between branches or can be updated by upstream at any point, one should install [https://github.com/bazelbuild/bazelisk Bazelisk]. It is a frontend to Bazel which automatically installs the needed Bazel version for the Gerrit branch you are working on. You will require the Go programming language in order to build it. The following will build Bazelisk 1.6.1 from source and copy the resulting binary in your user bin directory (<code>~/.local/bin</code>):<syntaxhighlight lang="console">
$ which go || sudo apt install golang-go
$ mkdir /tmp/bazelisk-build
$ (/tmp/bazelisk-build && go mod init . && go get github.com/bazelbuild/bazelisk@v1.6.1)
$ mkdir -p "$HOME"/.local/bin && cp /tmp/bazelisk-build/gopath/bin/bazelisk "$HOME"/.local/bin/bazelisk
$ rm -fR /tmp/bazelisk-build
</syntaxhighlight>

== Update our repository ==

=== Upgrade Gerrit core branch ===
Even if we deploy Gerrit core from the upstream released .war, we still need an up-to-date Gerrit copy in order to build our plugins.
#Clone Gerrit and submodules from Wikimedia, add upstream as a remote:<syntaxhighlight lang="bash">
git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
cd gerrit
git remote add upstream https://gerrit.googlesource.com/gerrit
</syntaxhighlight>
#Our repository <code>HEAD</code> points to the branch holding the material for the deployment <code>deploy/wmf/stable-3.2</code>. Checkout the <code>stable-3.2</code> branch (as of November 2020) which is a copy of the upstream release we currently use:<syntaxhighlight lang="bash">
git checkout stable-3.2 origin/stable-3.2
</syntaxhighlight>
# Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the <code>stable-3.2</code> directly to Gerrit:<syntaxhighlight lang="bash">
git fetch --tags upstream stable-3.2
git merge --ff-only v3.2.10
git push --tags origin HEAD:stable-3.2
</syntaxhighlight>
# Checkout <code>wmf/stable-3.2</code> and merge the latest tag, push the resulting '''SINGLE''' merge commit up for review:<syntaxhighlight lang="bash">
git checkout wmf/stable-3.2
git merge v3.2.10
git commit --amend --no-edit  # add Change-Id
git push origin HEAD:refs/for/wmf/stable-3.2
</syntaxhighlight>

=== Update bundled submodules ===
Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The <code>.gitmodules</code> file comes from upstream and uses '''relative urls'''. Since your workspace copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will would fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting <code>url.<base>.insteadOf = <our url></code>.

#Ensure you are in our forked branch:<syntaxhighlight lang="bash">
git checkout wmf/stable-3.2
# update as needed
</syntaxhighlight>
#Update git submodules from the upstream repository using: <syntaxhighlight lang="bash">
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
</syntaxhighlight>
#Push for review. At this point, you will likely be ahead of <code>wmf/stable-3.2</code> by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:<syntaxhighlight lang="bash">
git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
git push origin HEAD:refs/for/wmf/stable-3.2

</syntaxhighlight>

== Fetch our additional plugins ==
As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered as submodules in our branch. They will need to be updated to match the new Gerrit version. You will need <code>ssh</code> admin access to Gerrit to be able to list the installed plugins as well as <code>jq</code> (<code>sudo apt install jq</code>).
List our plugins:<syntaxhighlight lang="console">
$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$ 
</syntaxhighlight>

Most plugins come with a set of <code>stable-</code> branches, some support multiple versions and solely have a <code>master</code> branch. The proper branch to use for our extra plugins is indicated in  <code>.gitmodules</code> , we thus just have to update them from the remote branch (<code>git submodule update --remote plugins/foo</code>). To update our plugins:<syntaxhighlight lang="console">
$ ./wmf-plugins-update.sh
Updating Wikimedia specific plugins
<the list of our plugins is shown here>
<git submodule update if relevant>
Done
$ 
</syntaxhighlight>

Plugins which we have forked on our Gerrit such as <code>gitiles</code> need the fork to be updated and upstream tag to be pushed:
<code>
cd plugins/gitiles
git remote add google https://gerrit.googlesource.com/plugins/gitiles
git fetch google
git merge google/stable-3.2
git commit --amend --no-edit  # add Change-Id
</code>
<code>
git remote set-url --push origin ssh://gerrit.wikimedia.org:29418/operations/software/gerrit/plugins/gitiles.git
git push --tags origin HEAD:stable-3.2
</code>

Once done updating our plugins submodules, git add them, update your commit and send it for review:<syntaxhighlight lang="bash">
git add $(./wmf-plugins-list.sh)
git commit --amend
git push origin HEAD/refs/for/wmf/stable-3.2
</syntaxhighlight>CI will build Gerrit core as well as all the extra plugins, the resulting <code>.war</code> and <code>.jar</code> files are attached to the build.

== Build ==
For Gerrit core and its bundled plugins, just use the upstream released .war. This instructions are left here for reference in case we have to build Gerrit ourselves at some point although the logic has been ported to a Jenkins job and CI will happily build Gerrit and the plugins for you.

=== Gerrit core ===

==== Upstream .war ====

The gerrit.war can be retrieved simply by running <code>mvn package</code>:
<syntaxhighlight lang="bash">
$ mvn package
...
[INFO] --- maven-dependency-plugin:2.8:copy (copy) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.2.7:war
[INFO] com.google.gerrit:gerrit-war:3.2.7:war already exists in /home/hashar/projects/operations/software/gerrit
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
</syntaxhighlight>

That downloads the Gerrit war released by Google on Maven Central and replace the <code>/gerrit.war</code> at the root of the repository.

To update the Gerrit version, edit <code>pom.xml</code> and adjust the <code>gerrit.version</code> property:
<syntaxhighlight lang="xml">
  <properties>
    <gerrit.version>3.2.7</gerrit.version>
  </properties>
</syntaxhighlight>

Then run mvn, add the result and craft the commit:
<syntaxhighlight lang="console">
$ mvn package
...
$ git add pom.xml gerrit.war
$ git commit -m 'Update Gerrit to vX.Y.Z'
</syntaxhighlight>


==== Build our own ====

''As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use the upstream <code>.war</code>, see previous section. The build instructions are left there in case we have to build a fork.''

{{collapse top|Build instructions}}

# Checkout the <code>wmf/stable-3.2</code> (as of November 2020)<syntaxhighlight lang="bash">
git checkout -b wmf/stable-3.2 origin/wmf/stable-3.2
</syntaxhighlight>

<s>Production uses Java 8 as of November 19th 2020 which we will eventually switch to Java 11 ( https://phabricator.wikimedia.org/T268225 ). Meantime, Bazel has to be pointed to your local Java 8 installation: <code>JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64</code>.</s>

# <s>Build the <code>bazel-bin/release.war</code></s><syntaxhighlight lang="bash">
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bazel build release
</syntaxhighlight>

<s>Note that if your building Gerrit v3.3 (or later) or master from 2020-10-06 (or later) and you're targeting Java 8, you'll also have to add the argument <code>--java_toolchain=//tools:error_prone_warnings_toolchain</code> (this is untested, but claimed on the [https://groups.google.com/d/msgid/repo-discuss/1249A338-D5CF-4F3D-811B-8FEAA49AFE10%40gmail.com gerrit mailing list]).</s> 

{{collapse bottom}}

=== Plugins ===
Most plugins come with build instructions usually available at <code>src/main/resources/Documentation/build.md</code>.

Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. <code>wmf-plugins-list.sh</code> can list our plugins:
<syntaxhighlight lang="console">
$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$ 
</syntaxhighlight>

The whole build logic is implemented in the <code>wmf-build.py</code> script:
<syntaxhighlight lang="console">
$ ./wmf-build.py
Building Wikimedia specific plugins

> Building go-import

...

Done building
Copying build artifacts
plugins-wmf/go-import.jar
plugins-wmf/healthcheck.jar
plugins-wmf/its-base.jar
plugins-wmf/its-phabricator.jar
plugins-wmf/javamelody.jar
plugins-wmf/lfs.jar
plugins-wmf/metrics-reporter-jmx.jar
plugins-wmf/metrics-reporter-prometheus.jar
plugins-wmf/motd.jar
plugins-wmf/reviewers.jar
plugins-wmf/zuul.jar

Those should be copied to the wmf/deploy branch
Done
$ 
</syntaxhighlight>

Make sure to remove all content from <code>./plugins</code> before copying the build artifacts.

==== Add a new plugin ====

In our <code>wmf/stable-3.2</code> branch and the plugin upstream repository as a submodule. It should point to the current stable branch (<code>stable-3.2</code>) if supported, else <code>master</code>.
<syntaxhighlight lang="bash">
git submodule add -f -b stable-3.2 -- https://gerrit.googlesource.com/plugins/rename-project plugins/rename-project
git add .gitmodules plugins/rename-project
git commit -m 'Add rename-project plugin'
</syntaxhighlight>

Then follow the instructions from the previous section to build it and add the resulting <code>.jar</code> to our deployment branch.

If we were to fork a plugin, the process is similar.  The repository is forked on our Gerrit instance and then registered as a submodule pointing to our Gerrit.

== Upload artifacts ==
After you have imported Gerrit releases.war from upstream and build the plugins (<code>wmf-build.py</code>), the artifacts have to be added to the <code>deploy/wmf/stable-X</code> branch. Since they are rather large, we use <code>git-fat</code> to offload the git repository and those artifacts have to be uploaded to our [[Archiva]].

# Set up <code>git-fat</code>: [[Archiva#Setting_up_git-fat_for_your_project]]
# Initialization: <code>git fat init</code>
# Checkout the <code>deploy/wmf/stable-X</code> branch that aligns with the build branch
# Remove all the plugins in <code>./plugins</code>: <code>rm -f plugins/*.jar</code>

'''If you have build locally''':
#Copy all the previously build plugin jars from <code>./plugins-wmf</code> to <code>./plugins</code> in the deploy branch

'''If you want to rely on CI''':
# Head to the last merged change of the <code>wmf/stable-3.2</code> branch and follow the link to the [https://integration.wikimedia.org/ci/job/gerrit-docker/ gerrit-docker job build]
# On the build page follow the '''Build artifacts''' link
# Download a zip via '''(all files in zip)'''
# Extract the archive: <code>cd plugins && rm -f *.jar && unzip -o -j /path/to/archive.zip</code>
# Remove the release.war built by CI: <code>rm plugins/release.war</code>

Prepare the commit:

Ensure Gerrit bundled plugins '''are not included''' since they will be provided by the <code>releases.war</code>
* <code>rm plugins/its-base.jar</code>
Commit the changes and upload to Gerrit
* <code>git add plugins/*.jar</code>
* <code>git commit -m 'Update plugins for Gerrit 3.2.10'</code>
* <code>git push origin HEAD:refs/for/deploy/wmf/stable-3.2</code>


Run <code>./deploy_artifacts.py --version=3.2.10 gerrit.war plugins/foo.jar ...</code> for the core application and all updated plugins (cf: [[Archiva]], specifically setting up your <code>~/.m2/settings.xml</code>

; Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.

== Deploying ==
# SSH to the deployment master (<code>ssh deployment.eqiad.wmnet</code>)
# Navigate to <code>/srv/deployment/gerrit/gerrit</code>
# Fetch & checkout the appropriate <code>deploy/wmf/stable-X</code> branch
# Deploy to gerrit2001 to ensure that there are no errors with <code>git-fat</code> et al
#<code>scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'</code>
# Deploy to gerrit1001 <code>scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'</code>
# '''If you're only deploying plugins, you are done'''
When upgrading Gerrit <code>releases.war</code> to a new minor or major version, '''make sure to have taken in account upgrade steps that are mentioned in the upstream release notes.'''

Then on each server refresh the plugins from the <code>.war</code> and upgrade:

#<code>sudo service gerrit stop</code>
#<code>sudo -u gerrit2 java -jar /var/lib/gerrit2/review_site/bin/gerrit.war init --batch --install-all-plugins</code>
#<code>sudo -u gerrit2 git -C '/var/lib/gerrit2/review_site/plugins/' '*.jar'</code>
#<code>sudo service gerrit start</code>