You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Gerrit/Upgrade: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Hashar
m (→‎Fix-up of javamelody plugin: external plugins dependencies are now managed by the build script)
imported>Hashar
m (→‎Upload artifacts: upload_artifacts.py > deploy_artifacts.py)
Line 5: Line 5:


== Build prerequisites ==
== Build prerequisites ==
''The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.2/Documentation/dev-bazel.txt''
Install OpenJDK 8, git, NodeJS, Python 3. On Debian:
Install OpenJDK 8, git, NodeJS, Python 3. On Debian:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
Line 19: Line 22:


=== Upgrade Gerrit core branch ===
=== Upgrade Gerrit core branch ===
Even if we deploy Gerrit core from the upstream released .war, we still need an up-to-date Gerrit copy in order to build our plugins.
#Clone Gerrit and submodules from Wikimedia, add upstream as a remote:<syntaxhighlight lang="bash">
#Clone Gerrit and submodules from Wikimedia, add upstream as a remote:<syntaxhighlight lang="bash">
git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
cd gerrit
cd gerrit
git remote add upstream https://gerrit.googlesource.com/gerrit
git remote add upstream https://gerrit.googlesource.com/gerrit
</syntaxhighlight>Our repository <code>HEAD</code> points to the branch holding the material for the deployment <code>deploy/wmf/stable-3.2</code>
</syntaxhighlight>
#Checkout the <code>stable-3.2</code> branch (as of November 2020) which is a copy of the upstream release we currently use:<syntaxhighlight lang="bash">
#Our repository <code>HEAD</code> points to the branch holding the material for the deployment <code>deploy/wmf/stable-3.2</code>. Checkout the <code>stable-3.2</code> branch (as of November 2020) which is a copy of the upstream release we currently use:<syntaxhighlight lang="bash">
git checkout stable-3.2 origin/stable-3.2
git checkout stable-3.2 origin/stable-3.2
</syntaxhighlight>
</syntaxhighlight>
Line 38: Line 42:


=== Update bundled submodules ===
=== Update bundled submodules ===
Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The <code>.gitmodules</code> file comes from upstream and uses '''relative urls'''. Since your workspace copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting <code>url.<base>.insteadOf = <our url></code>.
Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The <code>.gitmodules</code> file comes from upstream and uses '''relative urls'''. Since your workspace copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will would fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting <code>url.<base>.insteadOf = <our url></code>.


#Ensure you are in our forked branch:<syntaxhighlight lang="bash">
#Ensure you are in our forked branch:<syntaxhighlight lang="bash">
Line 44: Line 48:
# update as needed
# update as needed
</syntaxhighlight>
</syntaxhighlight>
#Update git submodules from the upstream repository: <syntaxhighlight lang="bash">
#Update git submodules from the upstream repository using: <syntaxhighlight lang="bash">
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
</syntaxhighlight>
</syntaxhighlight>
# Push for review. At this point, you will likely be ahead of <code>wmf/stable-3.2</code> by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:<syntaxhighlight lang="bash">
#Push for review. At this point, you will likely be ahead of <code>wmf/stable-3.2</code> by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:<syntaxhighlight lang="bash">
git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
git push origin HEAD:refs/for/wmf/stable-3.2
git push origin HEAD:refs/for/wmf/stable-3.2
Line 54: Line 58:


== Fetch our additional plugins ==
== Fetch our additional plugins ==
As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered and they need to be updated to match the new Gerrit core. You will need <code>ssh</code> admin access to Gerrit to be able to list the installed plugins as well as <code>jq</code> (<code>sudo apt install jq</code>).
As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered as submodules in our branch. They will need to be updated to match the new Gerrit version. You will need <code>ssh</code> admin access to Gerrit to be able to list the installed plugins as well as <code>jq</code> (<code>sudo apt install jq</code>).
List our plugins:<syntaxhighlight lang="console">
List our plugins:<syntaxhighlight lang="console">
$ ./wmf-plugins-list.sh  
$ ./wmf-plugins-list.sh  
Line 70: Line 74:
$  
$  
</syntaxhighlight>
</syntaxhighlight>


Most plugins come with a set of <code>stable-</code> branches, some support multiple versions and solely have a <code>master</code> branch. The proper branch to use for our extra plugins is indicated in  <code>.gitmodules</code> , we thus just have to update them from the remote branch (<code>git submodule update --remote plugins/foo</code>). To update our plugins:<syntaxhighlight lang="console">
Most plugins come with a set of <code>stable-</code> branches, some support multiple versions and solely have a <code>master</code> branch. The proper branch to use for our extra plugins is indicated in  <code>.gitmodules</code> , we thus just have to update them from the remote branch (<code>git submodule update --remote plugins/foo</code>). To update our plugins:<syntaxhighlight lang="console">
Line 87: Line 90:


To achieve that, make sure your gitiles code includes [https://gerrit-review.googlesource.com/c/plugins/gitiles/+/273397 change I4f4a0b7dd575cbc27641063e05d13b8a43a51d8b]. The change did not get much traction upstream, so it probably won't land there. We nonetheless want to include that change.
To achieve that, make sure your gitiles code includes [https://gerrit-review.googlesource.com/c/plugins/gitiles/+/273397 change I4f4a0b7dd575cbc27641063e05d13b8a43a51d8b]. The change did not get much traction upstream, so it probably won't land there. We nonetheless want to include that change.


Once done updating our plugins submodules, git add them, update your commit and send it for review:<syntaxhighlight lang="bash">
Once done updating our plugins submodules, git add them, update your commit and send it for review:<syntaxhighlight lang="bash">
Line 93: Line 95:
git commit --amend
git commit --amend
git push origin HEAD/refs/for/wmf/stable-3.2
git push origin HEAD/refs/for/wmf/stable-3.2
</syntaxhighlight>
</syntaxhighlight>CI will build Gerrit core as well as all the extra plugins, the resulting <code>.war</code> and <code>.jar</code> files are attached to the build.


== Build Gerrit ==
== Build Gerrit ==
For Gerrit core and its bundled plugins, just use the upstream released .war. This instructions are left here for reference in case we have to build Gerrit ourselves at some point although the logic has been ported to a Jenkins job and CI will happily build Gerrit and the plugins for you.


=== Gerrit core ===
=== Gerrit core ===
Line 114: Line 117:
Most plugins come with build instructions usually available at <code>src/main/resources/Documentation/build.html</code>.
Most plugins come with build instructions usually available at <code>src/main/resources/Documentation/build.html</code>.


Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. The <code>wmf-plugins-build.sh</code> script iterate over the plugins found via <code>wmf-plugins-list.sh</code> and build them, taking care of dependencies files if needed:<syntaxhighlight lang="console">
Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. <code>wmf-plugins-list.sh</code> can list our plugins:<syntaxhighlight lang="console">
$ ./wmf-plugins-list.sh  
$ ./wmf-plugins-list.sh  
plugins/go-import
plugins/go-import
Line 129: Line 132:
$  
$  
</syntaxhighlight>
</syntaxhighlight>
Build them:<syntaxhighlight lang="console">
The whole build logic is implemented in the <code>wmf-build.py</code> script:<syntaxhighlight lang="console">
$ ./wmf-ci.py
$ ./wmf-build.py
Building Wikimedia specific plugins
Building Wikimedia specific plugins


Line 152: Line 155:


Those should be copied to the wmf/deploy branch
Those should be copied to the wmf/deploy branch
 
Done
$  
$  
</syntaxhighlight>
</syntaxhighlight>


== Upload artifacts ==
== Upload artifacts ==
# Checkout the <code>deploy/wmf/stable-X</code> branch that aligns with the build branch
After you have imported Gerrit releases.war from upstream and build the plugins (<code>wmf-build.py</code>), the artifacts have to be added to the <code>deploy/wmf/stable-X</code> branch. Since they are rather large, we use <code>git-fat</code> to offload the git repository and those artifacts have to be uploaded to our [[Archiva]].
# Copy all the plugin jars from <code>./plugins-wmf</code>over to <code>./plugins</code> in the deploy branch
#Checkout the <code>deploy/wmf/stable-X</code> branch that aligns with the build branch
#Ensure Gerrit bundled plugins are not included and that any plugins we no more have installed are removed
# Remove all the plugins in <code>./plugins</code>
#Copy all the previously build plugin jars from <code>./plugins-wmf</code> to <code>./plugins</code> in the deploy branch
#Ensure Gerrit bundled plugins '''are not included''' (they are in the <code>releases.war</code>)
# Commit the changes and upload to Gerrit
# Commit the changes and upload to Gerrit
# Run <code>./upload_artifacts.py --version=3.2.8 gerrit.war plugins/foo.jar ...</code> for the core application and all updated plugins (cf: [[Archiva]], specifically setting up your <code>~/.m2/settings.xml</code>
# Run <code>./deploy_artifacts.py --version=3.2.10 gerrit.war plugins/foo.jar ...</code> for the core application and all updated plugins (cf: [[Archiva]], specifically setting up your <code>~/.m2/settings.xml</code>
#: '''Note''': Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.
#: '''Note''': Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.


Line 168: Line 173:
# Navigate to <code>/srv/deployment/gerrit/gerrit</code>
# Navigate to <code>/srv/deployment/gerrit/gerrit</code>
# Fetch & checkout the appropriate <code>deploy/wmf/stable-X</code> branch
# Fetch & checkout the appropriate <code>deploy/wmf/stable-X</code> branch
# Deploy to gerrit2001 to ensure that there are no errors with git-fat et al <code>scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'</code>
# Deploy to gerrit2001 to ensure that there are no errors with <code>git-fat</code> et al
#<code>scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'</code>
# Deploy to gerrit1001 <code>scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'</code>
# Deploy to gerrit1001 <code>scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'</code>
# If you're only deploying plugins, you're done, otherwise SSH to the Gerrit master and issue <code>sudo service gerrit restart</code>
# '''If you're only deploying plugins, you are done'''
XXX ensure plugins bundled in the war files get installed: <code>java -jar gerrit.war init --install-all-plugins</code>
When upgrading Gerrit <code>releases.war</code> to a new minor or major version, make sure to have taken in account upgrade steps that are mentioned in the release notes.
 
Then on each server refresh the plugins from the <code>.war</code> and upgrade:


XXX add a warning that init would cause upgrade steps to be conducted if need be (aka from a Gerrit version to another)
#<code>sudo -u gerrit2 java -jar /var/lib/gerrit2/review_site/bin/gerrit.war init --install-all-plugins</code>
#Issue <code>sudo service gerrit restart</code>

Revision as of 15:56, 21 May 2021

Since Feb 2nd 2021, we deploy the upstream war file as is. It is fetched from https://www.gerritcodereview.com/ , stored using git-fat in operations/software/gerrit under a deploy/wmf/stable-X.Y branch. We still build extra plugins from sources, the build relies on the source code held in the Gerrit repository operations/software/gerrit in branch wmf/stable-X.Y

The upstream branches (master, stable-3.2) are manually kept in sync. We have a forked branch wmf/stable-3.2 which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository. Our branch can be used to cherry pick patches for plugins build purpose, but must eventually be send to upstream, reviewed and merged there.

Build prerequisites

The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.2/Documentation/dev-bazel.txt

Install OpenJDK 8, git, NodeJS, Python 3. On Debian:

$ sudo apt install git openjdk-8-jdk-headless nodejs python3

Gerrit and its plugins are build using Bazel. Since the exact version to use might differ between branches or can be updated by upstream at any point, one should install Bazelisk. It is a frontend to Bazel which automatically installs the needed Bazel version for the Gerrit branch you are working on. You will require the Go programming language in order to build it. The following will build Bazelisk 1.6.1 from source and copy the resulting binary in your user bin directory (~/.local/bin):

$ which go || sudo apt install golang-go
$ mkdir /tmp/bazelisk-build
$ (/tmp/bazelisk-build && go mod init . && go get github.com/bazelbuild/bazelisk@v1.6.1)
$ mkdir -p "$HOME"/.local/bin && cp /tmp/bazelisk-build/gopath/bin/bazelisk "$HOME"/.local/bin/bazelisk
$ rm -fR /tmp/bazelisk-build

Update our repository

Upgrade Gerrit core branch

Even if we deploy Gerrit core from the upstream released .war, we still need an up-to-date Gerrit copy in order to build our plugins.

  1. Clone Gerrit and submodules from Wikimedia, add upstream as a remote:
    git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
    cd gerrit
    git remote add upstream https://gerrit.googlesource.com/gerrit
    
  2. Our repository HEAD points to the branch holding the material for the deployment deploy/wmf/stable-3.2. Checkout the stable-3.2 branch (as of November 2020) which is a copy of the upstream release we currently use:
    git checkout stable-3.2 origin/stable-3.2
    
  3. Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the stable-3.2 directly to Gerrit:
    git fetch --tags upstream stable-3.2
    git merge --ff-only v3.2.10
    git push --tags origin HEAD:stable-3.2
    
  4. Checkout wmf/stable-3.2 and merge the latest tag, push the resulting SINGLE merge commit up for review:
    git checkout wmf/stable-3.2
    git merge v3.2.10
    

Update bundled submodules

Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The .gitmodules file comes from upstream and uses relative urls. Since your workspace copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will would fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting url.<base>.insteadOf = <our url>.

  1. Ensure you are in our forked branch:
    git checkout wmf/stable-3.2
    # update as needed
    
  2. Update git submodules from the upstream repository using:
    git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
    
  3. Push for review. At this point, you will likely be ahead of wmf/stable-3.2 by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:
    git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
    git push origin HEAD:refs/for/wmf/stable-3.2
    

Fetch our additional plugins

As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered as submodules in our branch. They will need to be updated to match the new Gerrit version. You will need ssh admin access to Gerrit to be able to list the installed plugins as well as jq (sudo apt install jq).

List our plugins:

$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$

Most plugins come with a set of stable- branches, some support multiple versions and solely have a master branch. The proper branch to use for our extra plugins is indicated in .gitmodules , we thus just have to update them from the remote branch (git submodule update --remote plugins/foo). To update our plugins:

$ ./wmf-plugins-update.sh
Updating Wikimedia specific plugins
<the list of our plugins is shown here>
<git submodule update if relevant>
Done
$

Fix-up of gitiles plugin

TODO we should fork it.

The gitiles plugin per default fetches fonts directly from Google. Since that's a third party dependency that we do not want, we instead want gitiles to use the fonts that Gerrit itself offers.

To achieve that, make sure your gitiles code includes change I4f4a0b7dd575cbc27641063e05d13b8a43a51d8b. The change did not get much traction upstream, so it probably won't land there. We nonetheless want to include that change.

Once done updating our plugins submodules, git add them, update your commit and send it for review:

git add $(./wmf-plugins-list.sh)
git commit --amend
git push origin HEAD/refs/for/wmf/stable-3.2

CI will build Gerrit core as well as all the extra plugins, the resulting .war and .jar files are attached to the build.

Build Gerrit

For Gerrit core and its bundled plugins, just use the upstream released .war. This instructions are left here for reference in case we have to build Gerrit ourselves at some point although the logic has been ported to a Jenkins job and CI will happily build Gerrit and the plugins for you.

Gerrit core

As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use upstream .war.

  1. Checkout the wmf/stable-3.2 (as of November 2020)
    git checkout -b wmf/stable-3.2 origin/wmf/stable-3.2
    

Production uses Java 8 as of November 19th 2020 which we will eventually switch to Java 11 ( https://phabricator.wikimedia.org/T268225 ). Meantime, Bazel has to be pointed to your local Java 8 installation: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64.

  1. Build the bazel-bin/release.war
    JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bazel build release
    

Note that if your building Gerrit v3.3 (or later) or master from 2020-10-06 (or later) and you're targeting Java 8, you'll also have to add the argument --java_toolchain=//tools:error_prone_warnings_toolchain (this is untested, but claimed on the gerrit mailing list).

Plugins

Most plugins come with build instructions usually available at src/main/resources/Documentation/build.html.

Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. wmf-plugins-list.sh can list our plugins:

$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$

The whole build logic is implemented in the wmf-build.py script:

$ ./wmf-build.py
Building Wikimedia specific plugins

> Building go-import

...

Done building
Copying build artifacts
plugins-wmf/go-import.jar
plugins-wmf/healthcheck.jar
plugins-wmf/its-base.jar
plugins-wmf/its-phabricator.jar
plugins-wmf/javamelody.jar
plugins-wmf/lfs.jar
plugins-wmf/metrics-reporter-jmx.jar
plugins-wmf/metrics-reporter-prometheus.jar
plugins-wmf/motd.jar
plugins-wmf/reviewers.jar
plugins-wmf/zuul.jar

Those should be copied to the wmf/deploy branch
Done
$

Upload artifacts

After you have imported Gerrit releases.war from upstream and build the plugins (wmf-build.py), the artifacts have to be added to the deploy/wmf/stable-X branch. Since they are rather large, we use git-fat to offload the git repository and those artifacts have to be uploaded to our Archiva.

  1. Checkout the deploy/wmf/stable-X branch that aligns with the build branch
  2. Remove all the plugins in ./plugins
  3. Copy all the previously build plugin jars from ./plugins-wmf to ./plugins in the deploy branch
  4. Ensure Gerrit bundled plugins are not included (they are in the releases.war)
  5. Commit the changes and upload to Gerrit
  6. Run ./deploy_artifacts.py --version=3.2.10 gerrit.war plugins/foo.jar ... for the core application and all updated plugins (cf: Archiva, specifically setting up your ~/.m2/settings.xml
    Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.

Deploying

  1. SSH to the deployment master (ssh deployment.eqiad.wmnet)
  2. Navigate to /srv/deployment/gerrit/gerrit
  3. Fetch & checkout the appropriate deploy/wmf/stable-X branch
  4. Deploy to gerrit2001 to ensure that there are no errors with git-fat et al
  5. scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'
  6. Deploy to gerrit1001 scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'
  7. If you're only deploying plugins, you are done

When upgrading Gerrit releases.war to a new minor or major version, make sure to have taken in account upgrade steps that are mentioned in the release notes.

Then on each server refresh the plugins from the .war and upgrade:

  1. sudo -u gerrit2 java -jar /var/lib/gerrit2/review_site/bin/gerrit.war init --install-all-plugins
  2. Issue sudo service gerrit restart