You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Gerrit/Upgrade: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Hashar
imported>Hashar
(links to resume Icinga monitoring)
 
(5 intermediate revisions by 2 users not shown)
Line 2: Line 2:
Since Feb 2nd 2021, we deploy the upstream war file as is. It is fetched from https://www.gerritcodereview.com/ , stored using git-fat in {{git repo|operations/software/gerrit}} under a <code>deploy/wmf/stable-X.Y</code> branch. We still build extra plugins from sources, the build relies on the source code held in the Gerrit repository {{git repo|operations/software/gerrit}} in branch <code>wmf/stable-X.Y</code>
Since Feb 2nd 2021, we deploy the upstream war file as is. It is fetched from https://www.gerritcodereview.com/ , stored using git-fat in {{git repo|operations/software/gerrit}} under a <code>deploy/wmf/stable-X.Y</code> branch. We still build extra plugins from sources, the build relies on the source code held in the Gerrit repository {{git repo|operations/software/gerrit}} in branch <code>wmf/stable-X.Y</code>


The upstream branches (<code>master</code>, <code>stable-3.2</code>) are manually kept in sync. We have a forked branch <code>wmf/stable-3.2</code> which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository. Our branch can be used to cherry pick patches for plugins build purpose, but must eventually be send to upstream, reviewed and merged there. The branch overview:
The upstream branches (<code>master</code>, <code>stable-3.3</code>) are manually kept in sync. We have a forked branch <code>wmf/stable-3.3</code> which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository. Our branch can be used to cherry pick patches for plugins build purpose, but must eventually be send to upstream, reviewed and merged there. The branch overview:


{| class="wikitable"
{| class="wikitable"
Line 8: Line 8:
! Branch !! Description
! Branch !! Description
|-
|-
| <code>stable-3.2</code> || Gerrit core, updated from upstream
| <code>stable-3.3</code> || Gerrit core, updated from upstream
|-
|-
| <code>wmf/stable-3.2</code> || Fork holding our build script and our plugins as submodules
| <code>wmf/stable-3.3</code> || Fork holding our build script and our plugins as submodules
|-
|-
| <code>wmf/deploy/stable-3.2</code> || scap deployment repository holding Gerrit .war and plugins .jar with git-fat
| <code>wmf/deploy/stable-3.3</code> || scap deployment repository holding Gerrit .war and plugins .jar with git-fat
|}
|}


== Build prerequisites ==
== Build prerequisites ==


''The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.2/Documentation/dev-bazel.txt''
''The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.3/Documentation/dev-bazel.txt''


Install OpenJDK 8, git, NodeJS, Python 3. On Debian:
Install OpenJDK 8, git, NodeJS, Python 3. On Debian:
Line 39: Line 39:
git remote add upstream https://gerrit.googlesource.com/gerrit
git remote add upstream https://gerrit.googlesource.com/gerrit
</syntaxhighlight>
</syntaxhighlight>
#Our repository <code>HEAD</code> points to the branch holding the material for the deployment <code>deploy/wmf/stable-3.2</code>. Checkout the <code>stable-3.2</code> branch (as of November 2020) which is a copy of the upstream release we currently use:<syntaxhighlight lang="bash">
#Our repository <code>HEAD</code> points to the branch holding the material for the deployment <code>deploy/wmf/stable-3.3</code>. Checkout the <code>stable-3.3</code> branch (as of November 2020) which is a copy of the upstream release we currently use:<syntaxhighlight lang="bash">
git checkout stable-3.2 origin/stable-3.2
git checkout stable-3.3 origin/stable-3.3
</syntaxhighlight>
</syntaxhighlight>
# Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the <code>stable-3.2</code> directly to Gerrit:<syntaxhighlight lang="bash">
# Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the <code>stable-3.3</code> directly to Gerrit:<syntaxhighlight lang="bash">
git fetch --tags upstream stable-3.2
git fetch --tags upstream stable-3.3
git merge --ff-only v3.2.10
git merge --ff-only v3.3.6
git push --tags origin HEAD:stable-3.2
git push --tags origin HEAD:stable-3.3
</syntaxhighlight>
</syntaxhighlight>
# Checkout <code>wmf/stable-3.2</code> and merge the latest tag, push the resulting '''SINGLE''' merge commit up for review:<syntaxhighlight lang="bash">
# Checkout <code>wmf/stable-3.3</code> and merge the latest tag, push the resulting '''SINGLE''' merge commit up for review:<syntaxhighlight lang="bash">
git checkout wmf/stable-3.2
git checkout wmf/stable-3.3
git merge v3.2.10
git merge v3.3.6
git commit --amend --no-edit  # add Change-Id
git commit --amend --no-edit  # add Change-Id
git push origin HEAD:refs/for/wmf/stable-3.2
git push origin HEAD:refs/for/wmf/stable-3.3
</syntaxhighlight>
</syntaxhighlight>


Line 58: Line 58:


#Ensure you are in our forked branch:<syntaxhighlight lang="bash">
#Ensure you are in our forked branch:<syntaxhighlight lang="bash">
git checkout wmf/stable-3.2
git checkout wmf/stable-3.3
# update as needed
# update as needed
</syntaxhighlight>
</syntaxhighlight>
Line 64: Line 64:
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
</syntaxhighlight>
</syntaxhighlight>
#Push for review. At this point, you will likely be ahead of <code>wmf/stable-3.2</code> by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:<syntaxhighlight lang="bash">
#Push for review. At this point, you will likely be ahead of <code>wmf/stable-3.3</code> by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:<syntaxhighlight lang="bash">
git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
git push origin HEAD:refs/for/wmf/stable-3.2
git push origin HEAD:refs/for/wmf/stable-3.3


</syntaxhighlight>
</syntaxhighlight>
Line 102: Line 102:
git remote add google https://gerrit.googlesource.com/plugins/gitiles
git remote add google https://gerrit.googlesource.com/plugins/gitiles
git fetch google
git fetch google
git merge google/stable-3.2
git merge google/stable-3.3
git commit --amend --no-edit  # add Change-Id
git commit --amend --no-edit  # add Change-Id
</code>
</code>
<code>
<code>
git remote set-url --push origin ssh://gerrit.wikimedia.org:29418/operations/software/gerrit/plugins/gitiles.git
git remote set-url --push origin ssh://gerrit.wikimedia.org:29418/operations/software/gerrit/plugins/gitiles.git
git push --tags origin HEAD:stable-3.2
git push --tags origin HEAD:stable-3.3
</code>
</code>


Line 113: Line 113:
git add $(./wmf-plugins-list.sh)
git add $(./wmf-plugins-list.sh)
git commit --amend
git commit --amend
git push origin HEAD/refs/for/wmf/stable-3.2
git push origin HEAD/refs/for/wmf/stable-3.3
</syntaxhighlight>CI will build Gerrit core as well as all the extra plugins, the resulting <code>.war</code> and <code>.jar</code> files are attached to the build.
</syntaxhighlight>CI will build Gerrit core as well as all the extra plugins, the resulting <code>.war</code> and <code>.jar</code> files are attached to the build.


Line 125: Line 125:
The gerrit.war can be retrieved simply by running <code>mvn package</code>:
The gerrit.war can be retrieved simply by running <code>mvn package</code>:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
$ git checkout deploy/wmf/stable-3.3
$ mvn package
$ mvn package
...
...
[INFO] --- maven-dependency-plugin:2.8:copy (copy) @ gerrit ---
[INFO] --- maven-dependency-plugin:2.8:copy (copy) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.2.7:war
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.3.6:war
[INFO] com.google.gerrit:gerrit-war:3.2.7:war already exists in /home/hashar/projects/operations/software/gerrit
[INFO] com.google.gerrit:gerrit-war:3.3.6:war already exists in /home/hashar/projects/operations/software/gerrit
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] BUILD SUCCESS
Line 140: Line 141:
<syntaxhighlight lang="xml">
<syntaxhighlight lang="xml">
   <properties>
   <properties>
     <gerrit.version>3.2.7</gerrit.version>
     <gerrit.version>3.3.6</gerrit.version>
   </properties>
   </properties>
</syntaxhighlight>
</syntaxhighlight>
Line 147: Line 148:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ mvn package
$ mvn package
...
[INFO] --- maven-dependency-plugin:2.8:copy (copy) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.3.9:war
Downloading from wikimedia.mirrored: https://archiva.wikimedia.org/repository/mirrored/com/google/gerrit/gerrit-war/3.3.9/gerrit-war-3.3.9.war
...
...
$ git add pom.xml gerrit.war
$ git add pom.xml gerrit.war
$ git commit -m 'Update Gerrit to vX.Y.Z'
$ git commit -m 'Update Gerrit to vX.Y.Z'
</syntaxhighlight>
</syntaxhighlight>


==== Build our own ====
==== Build our own ====
Line 159: Line 163:
{{collapse top|Build instructions}}
{{collapse top|Build instructions}}


# Checkout the <code>wmf/stable-3.2</code> (as of November 2020)<syntaxhighlight lang="bash">
# Checkout the <code>wmf/stable-3.3</code> (as of November 2020)<syntaxhighlight lang="bash">
git checkout -b wmf/stable-3.2 origin/wmf/stable-3.2
git checkout -b wmf/stable-3.3 origin/wmf/stable-3.3
</syntaxhighlight>
</syntaxhighlight>


Line 178: Line 182:
Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. <code>wmf-plugins-list.sh</code> can list our plugins:
Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. <code>wmf-plugins-list.sh</code> can list our plugins:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ git checkout wmf/stable-3.3
$ ./wmf-plugins-list.sh  
$ ./wmf-plugins-list.sh  
plugins/go-import
plugins/go-import
Line 195: Line 200:
The whole build logic is implemented in the <code>wmf-build.py</code> script:
The whole build logic is implemented in the <code>wmf-build.py</code> script:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ git checkout wmf/stable-3.3
$ ./wmf-build.py
$ ./wmf-build.py
Building Wikimedia specific plugins
Building Wikimedia specific plugins
Line 225: Line 231:
==== Add a new plugin ====
==== Add a new plugin ====


In our <code>wmf/stable-3.2</code> branch and the plugin upstream repository as a submodule. It should point to the current stable branch (<code>stable-3.2</code>) if supported, else <code>master</code>.
In our <code>wmf/stable-3.3</code> branch and the plugin upstream repository as a submodule. It should point to the current stable branch (<code>stable-3.3</code>) if supported, else <code>master</code>.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
git submodule add -f -b stable-3.2 -- https://gerrit.googlesource.com/plugins/rename-project plugins/rename-project
git submodule add -f -b stable-3.3 -- https://gerrit.googlesource.com/plugins/rename-project plugins/rename-project
git add .gitmodules plugins/rename-project
git add .gitmodules plugins/rename-project
git commit -m 'Add rename-project plugin'
git commit -m 'Add rename-project plugin'
Line 248: Line 254:


'''If you want to rely on CI''':
'''If you want to rely on CI''':
# Head to the last merged change of the <code>wmf/stable-3.2</code> branch and follow the link to the [https://integration.wikimedia.org/ci/job/gerrit-docker/ gerrit-docker job build]
# Head to the last merged change of the <code>wmf/stable-3.3</code> branch and follow the link to the [https://integration.wikimedia.org/ci/job/gerrit-docker/ gerrit-docker job build]
# On the build page follow the '''Build artifacts''' link
# On the build page follow the '''Build artifacts''' link
# Download a zip via '''(all files in zip)'''
# Download a zip via '''(all files in zip)'''
Line 259: Line 265:
* <code>rm plugins/its-base.jar</code>
* <code>rm plugins/its-base.jar</code>
Commit the changes and upload to Gerrit
Commit the changes and upload to Gerrit
* <code>git add plugins/*.jar</code>
* <code>git add 'plugins/*.jar'</code>
* <code>git commit -m 'Update plugins for Gerrit 3.2.10'</code>
* Note if a plugin is no more present,
* <code>git push origin HEAD:refs/for/deploy/wmf/stable-3.2</code>
* <code>git commit -m 'Update plugins for Gerrit 3.3.6'</code>
* <code>git push origin HEAD:refs/for/deploy/wmf/stable-3.3</code>




Run <code>./deploy_artifacts.py --version=3.2.10 gerrit.war plugins/foo.jar ...</code> for the core application and all updated plugins (cf: [[Archiva]], specifically setting up your <code>~/.m2/settings.xml</code>
Run <code>./deploy_artifacts.py --version=3.3.6 gerrit.war plugins/foo.jar ...</code> for the core application and all updated plugins (cf: [[Archiva]], specifically setting up your <code>~/.m2/settings.xml</code>


; Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.
; Note: Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.
Line 270: Line 277:
== Deploying ==
== Deploying ==


Schedule downtime for Gerrit hosts to avoid notifications:
NOTE: when upgrading Gerrit <code>releases.war</code> to a new minor or major version: '''make sure to have taken in account upgrade steps that are mentioned in the upstream release notes.'''
 
1) Schedule downtime for Gerrit hosts to avoid notifications
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=86&host=gerrit1001
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=86&host=gerrit1001
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=86&host=gerrit2001
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=86&host=gerrit2002
 
2) Disable Puppet
 
This make sure Puppet does not restart the service or alter the <code>gerrit.config</code>:
<syntaxhighlight lang="shell">sudo puppet agent --disable 'Gerrit deployment'</syntaxhighlight>


Disable Puppet to prevent it from restarting the service:
3) Deploy artifacts
* <code>sudo puppet agent --disable 'Gerrit deployment'</code>


# SSH to the deployment master (<code>ssh deployment.eqiad.wmnet</code>)
# SSH to the deployment master (<code>ssh deployment.eqiad.wmnet</code>)
# Navigate to <code>/srv/deployment/gerrit/gerrit</code>
# <code>cd /srv/deployment/gerrit/gerrit</code>
# Update the repository: <code>git fetch</code>
# Checkout the appropriate <code>deploy/wmf/stable-X</code> branch: <code>git checkout deploy/wmf/stable-X</code>
# Checkout the appropriate <code>deploy/wmf/stable-X</code> branch: <code>git checkout deploy/wmf/stable-X</code>
# Update the repository: <code>git fetch</code>
# Review differences: <code>git log HEAD..HEAD@{u}</code>
# Review differences: <code>git log HEAD..HEAD@{u}</code>
# <code>git rebase</code> if all is fine
# <code>git rebase</code> if all is fine


Deploy to gerrit2001 to ensure that there are no errors with <code>git-fat</code> et al
Deploy artifacts to gerrit2002 to ensure that there are no errors with <code>git-fat</code> et al
#<code>scap deploy -l gerrit2001.wikimedia.org 'Gerrit to [version] on gerrit2001'</code>
#<code>scap deploy -l gerrit2002.wikimedia.org 'Gerrit to [version] on gerrit2002'</code>


Deploy to gerrit1001:
Deploy to gerrit1001:
# <code>scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'</code>
# <code>scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'</code>


'''If you're only deploying plugins, you are done'''
4) Refresh plugins


When upgrading Gerrit <code>releases.war</code> to a new minor or major version, '''make sure to have taken in account upgrade steps that are mentioned in the upstream release notes.'''
On each server refresh the plugins from the <code>.war</code> and upgrade:


Then on each server refresh the plugins from the <code>.war</code> and upgrade:
; Accessing servers
: <code>ssh gerrit2002.wikimedia.org</code> (replica)
: <code>ssh gerrit1001.wikimedia.org</code> (primary)


Accessing servers:
<code>ssh gerrit-replica.wikimedia.org</code> - this is gerrit2001
<code>ssh gerrit1001.wikimedia.org</code> - this is gerrit1001


TODO: put gerrit in maintenance mode
TODO: put gerrit in maintenance mode
TODO: have scap run commands
Extract plugins bundled in Gerrit core. Since we have forked some of the bundled plugins and Gerrit scans the <code>./plugins</code> directory and would reload/drop/add the plugins automatically, we need to <b>stop Gerrit</b>.


#<code>sudo service gerrit stop</code>
#<code>sudo service gerrit stop</code>
Line 307: Line 322:
#<code>java -jar bin/gerrit.war init --batch --install-all-plugins</code>
#<code>java -jar bin/gerrit.war init --batch --install-all-plugins</code>


Verify the plugins:
5) Run initialization
 
<code>gerrit init</code> extract the plugins from the upstream war, run their init steps, upgrade the schema and normalize <code>gerrit.config</code>.
 
For upstream bundled plugins we have forked (such as gitiles), we need to restore our version. That is done by restoring git tracked <code>.jar</code> from our deployment repository. Rest of bundled plugins are untracked in our git repository.
# <code>cd plugins</code>
# <code>cd plugins</code>
# <code>git status</code> - gitiles is modified, and a bunch of untracked .jar files
# <code>git status</code>  
# <code>git checkout '*.jar'</code>
# <code>git checkout '*.jar'</code> # restore OUR version of upstream bundled plugins
 
6) Apply our config
Enable and run Puppet:
<syntaxhighlight lang="shell">
sudo puppet agent --enable
sudo puppet agent -tv
</syntaxhighlight>
 
Take note of any change that might have been applied to <code>gerrit.config</code>: they should be transcribed back to Puppet.
 
7) Puppet will start up Gerrit for you.
 
8) Checks
 
Plugins either on the web https://gerrit.wikimedia.org/r/admin/plugins or via ssh:
* <code>ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls</code>
* <code>ssh -p 29418 gerrit-replica.wikimedia.org gerrit plugin ls</code>
 
 
<syntaxhighlight lang="shell">
sudo journalctl -u gerrit -n 50 --follow
/var/log/gerrit/error_log
</syntaxhighlight>


Restart the Gerrit daemon:
Logs are in Kibana https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV
# exit gerrit2 sudo session
#<code>sudo service gerrit start</code>


Enable puppet:
9) Resume monitoring
* <code>sudo puppet agent --enable
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=171&host=gerrit1001
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=171&host=gerrit2002

Latest revision as of 09:14, 17 August 2022

Since Feb 2nd 2021, we deploy the upstream war file as is. It is fetched from https://www.gerritcodereview.com/ , stored using git-fat in operations/software/gerrit under a deploy/wmf/stable-X.Y branch. We still build extra plugins from sources, the build relies on the source code held in the Gerrit repository operations/software/gerrit in branch wmf/stable-X.Y

The upstream branches (master, stable-3.3) are manually kept in sync. We have a forked branch wmf/stable-3.3 which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository. Our branch can be used to cherry pick patches for plugins build purpose, but must eventually be send to upstream, reviewed and merged there. The branch overview:

operations/software/gerrit branches layout
Branch Description
stable-3.3 Gerrit core, updated from upstream
wmf/stable-3.3 Fork holding our build script and our plugins as submodules
wmf/deploy/stable-3.3 scap deployment repository holding Gerrit .war and plugins .jar with git-fat

Build prerequisites

The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.3/Documentation/dev-bazel.txt

Install OpenJDK 8, git, NodeJS, Python 3. On Debian:

$ sudo apt install git openjdk-8-jdk-headless nodejs python3

Gerrit and its plugins are build using Bazel. Since the exact version to use might differ between branches or can be updated by upstream at any point, one should install Bazelisk. It is a frontend to Bazel which automatically installs the needed Bazel version for the Gerrit branch you are working on. You will require the Go programming language in order to build it. The following will build Bazelisk 1.6.1 from source and copy the resulting binary in your user bin directory (~/.local/bin):

$ which go || sudo apt install golang-go
$ mkdir /tmp/bazelisk-build
$ (/tmp/bazelisk-build && go mod init . && go get github.com/bazelbuild/bazelisk@v1.6.1)
$ mkdir -p "$HOME"/.local/bin && cp /tmp/bazelisk-build/gopath/bin/bazelisk "$HOME"/.local/bin/bazelisk
$ rm -fR /tmp/bazelisk-build

Update our repository

Upgrade Gerrit core branch

Even if we deploy Gerrit core from the upstream released .war, we still need an up-to-date Gerrit copy in order to build our plugins.

  1. Clone Gerrit and submodules from Wikimedia, add upstream as a remote:
    git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
    cd gerrit
    git remote add upstream https://gerrit.googlesource.com/gerrit
    
  2. Our repository HEAD points to the branch holding the material for the deployment deploy/wmf/stable-3.3. Checkout the stable-3.3 branch (as of November 2020) which is a copy of the upstream release we currently use:
    git checkout stable-3.3 origin/stable-3.3
    
  3. Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the stable-3.3 directly to Gerrit:
    git fetch --tags upstream stable-3.3
    git merge --ff-only v3.3.6
    git push --tags origin HEAD:stable-3.3
    
  4. Checkout wmf/stable-3.3 and merge the latest tag, push the resulting SINGLE merge commit up for review:
    git checkout wmf/stable-3.3
    git merge v3.3.6
    git commit --amend --no-edit  # add Change-Id
    git push origin HEAD:refs/for/wmf/stable-3.3
    

Update bundled submodules

Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The .gitmodules file comes from upstream and uses relative urls. Since your workspace copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will would fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting url.<base>.insteadOf = <our url>.

  1. Ensure you are in our forked branch:
    git checkout wmf/stable-3.3
    # update as needed
    
  2. Update git submodules from the upstream repository using:
    git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
    
  3. Push for review. At this point, you will likely be ahead of wmf/stable-3.3 by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your changes:
    git commit --ammend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
    git push origin HEAD:refs/for/wmf/stable-3.3
    

Fetch our additional plugins

As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered as submodules in our branch. They will need to be updated to match the new Gerrit version. You will need ssh admin access to Gerrit to be able to list the installed plugins as well as jq (sudo apt install jq).

List our plugins:

$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$

Most plugins come with a set of stable- branches, some support multiple versions and solely have a master branch. The proper branch to use for our extra plugins is indicated in .gitmodules , we thus just have to update them from the remote branch (git submodule update --remote plugins/foo). To update our plugins:

$ ./wmf-plugins-update.sh
Updating Wikimedia specific plugins
<the list of our plugins is shown here>
<git submodule update if relevant>
Done
$

Plugins which we have forked on our Gerrit such as gitiles need the fork to be updated and upstream tag to be pushed: cd plugins/gitiles git remote add google https://gerrit.googlesource.com/plugins/gitiles git fetch google git merge google/stable-3.3 git commit --amend --no-edit # add Change-Id git remote set-url --push origin ssh://gerrit.wikimedia.org:29418/operations/software/gerrit/plugins/gitiles.git git push --tags origin HEAD:stable-3.3

Once done updating our plugins submodules, git add them, update your commit and send it for review:

git add $(./wmf-plugins-list.sh)
git commit --amend
git push origin HEAD/refs/for/wmf/stable-3.3

CI will build Gerrit core as well as all the extra plugins, the resulting .war and .jar files are attached to the build.

Build

For Gerrit core and its bundled plugins, just use the upstream released .war. This instructions are left here for reference in case we have to build Gerrit ourselves at some point although the logic has been ported to a Jenkins job and CI will happily build Gerrit and the plugins for you.

Gerrit core

Upstream .war

The gerrit.war can be retrieved simply by running mvn package:

$ git checkout deploy/wmf/stable-3.3
$ mvn package
...
[INFO] --- maven-dependency-plugin:2.8:copy (copy) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.3.6:war
[INFO] com.google.gerrit:gerrit-war:3.3.6:war already exists in /home/hashar/projects/operations/software/gerrit
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

That downloads the Gerrit war released by Google on Maven Central and replace the /gerrit.war at the root of the repository.

To update the Gerrit version, edit pom.xml and adjust the gerrit.version property:

  <properties>
    <gerrit.version>3.3.6</gerrit.version>
  </properties>

Then run mvn, add the result and craft the commit:

$ mvn package
...
[INFO] --- maven-dependency-plugin:2.8:copy (copy) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.3.9:war
Downloading from wikimedia.mirrored: https://archiva.wikimedia.org/repository/mirrored/com/google/gerrit/gerrit-war/3.3.9/gerrit-war-3.3.9.war
...
$ git add pom.xml gerrit.war
$ git commit -m 'Update Gerrit to vX.Y.Z'

Build our own

As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use the upstream .war, see previous section. The build instructions are left there in case we have to build a fork.

Plugins

Most plugins come with build instructions usually available at src/main/resources/Documentation/build.md.

Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. wmf-plugins-list.sh can list our plugins:

$ git checkout wmf/stable-3.3
$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/motd
plugins/reviewers
plugins/zuul
$

The whole build logic is implemented in the wmf-build.py script:

$ git checkout wmf/stable-3.3
$ ./wmf-build.py
Building Wikimedia specific plugins

> Building go-import

...

Done building
Copying build artifacts
plugins-wmf/go-import.jar
plugins-wmf/healthcheck.jar
plugins-wmf/its-base.jar
plugins-wmf/its-phabricator.jar
plugins-wmf/javamelody.jar
plugins-wmf/lfs.jar
plugins-wmf/metrics-reporter-jmx.jar
plugins-wmf/metrics-reporter-prometheus.jar
plugins-wmf/motd.jar
plugins-wmf/reviewers.jar
plugins-wmf/zuul.jar

Those should be copied to the wmf/deploy branch
Done
$

Make sure to remove all content from ./plugins before copying the build artifacts.

Add a new plugin

In our wmf/stable-3.3 branch and the plugin upstream repository as a submodule. It should point to the current stable branch (stable-3.3) if supported, else master.

git submodule add -f -b stable-3.3 -- https://gerrit.googlesource.com/plugins/rename-project plugins/rename-project
git add .gitmodules plugins/rename-project
git commit -m 'Add rename-project plugin'

Then follow the instructions from the previous section to build it and add the resulting .jar to our deployment branch.

If we were to fork a plugin, the process is similar. The repository is forked on our Gerrit instance and then registered as a submodule pointing to our Gerrit.

Upload artifacts

After you have imported Gerrit releases.war from upstream and build the plugins (wmf-build.py), the artifacts have to be added to the deploy/wmf/stable-X branch. Since they are rather large, we use git-fat to offload the git repository and those artifacts have to be uploaded to our Archiva.

  1. Set up git-fat: Archiva#Setting_up_git-fat_for_your_project
  2. Initialization: git fat init
  3. Checkout the deploy/wmf/stable-X branch that aligns with the build branch
  4. Remove all the plugins in ./plugins: rm -f plugins/*.jar

If you have build locally:

  1. Copy all the previously build plugin jars from ./plugins-wmf to ./plugins in the deploy branch

If you want to rely on CI:

  1. Head to the last merged change of the wmf/stable-3.3 branch and follow the link to the gerrit-docker job build
  2. On the build page follow the Build artifacts link
  3. Download a zip via (all files in zip)
  4. Extract the archive: cd plugins && rm -f *.jar && unzip -o -j /path/to/archive.zip
  5. Remove the release.war built by CI: rm plugins/release.war

Prepare the commit:

Ensure Gerrit bundled plugins are not included since they will be provided by the releases.war

  • rm plugins/its-base.jar

Commit the changes and upload to Gerrit

  • git add 'plugins/*.jar'
  • Note if a plugin is no more present,
  • git commit -m 'Update plugins for Gerrit 3.3.6'
  • git push origin HEAD:refs/for/deploy/wmf/stable-3.3


Run ./deploy_artifacts.py --version=3.3.6 gerrit.war plugins/foo.jar ... for the core application and all updated plugins (cf: Archiva, specifically setting up your ~/.m2/settings.xml

Note
Alternatively, you can upload them by using the web UI, but that gets repetitive over a bunch of plugins hence why we wrote the above tool.

Deploying

NOTE: when upgrading Gerrit releases.war to a new minor or major version: make sure to have taken in account upgrade steps that are mentioned in the upstream release notes.

1) Schedule downtime for Gerrit hosts to avoid notifications

2) Disable Puppet

This make sure Puppet does not restart the service or alter the gerrit.config:

sudo puppet agent --disable 'Gerrit deployment'

3) Deploy artifacts

  1. SSH to the deployment master (ssh deployment.eqiad.wmnet)
  2. cd /srv/deployment/gerrit/gerrit
  3. Update the repository: git fetch
  4. Checkout the appropriate deploy/wmf/stable-X branch: git checkout deploy/wmf/stable-X
  5. Review differences: git log HEAD..HEAD@{u}
  6. git rebase if all is fine

Deploy artifacts to gerrit2002 to ensure that there are no errors with git-fat et al

  1. scap deploy -l gerrit2002.wikimedia.org 'Gerrit to [version] on gerrit2002'

Deploy to gerrit1001:

  1. scap deploy -l gerrit1001.wikimedia.org 'Gerrit to [version] on gerrit1001'

4) Refresh plugins

On each server refresh the plugins from the .war and upgrade:

Accessing servers
ssh gerrit2002.wikimedia.org (replica)
ssh gerrit1001.wikimedia.org (primary)


TODO: put gerrit in maintenance mode TODO: have scap run commands

Extract plugins bundled in Gerrit core. Since we have forked some of the bundled plugins and Gerrit scans the ./plugins directory and would reload/drop/add the plugins automatically, we need to stop Gerrit.

  1. sudo service gerrit stop
  2. sudo -u gerrit2 -i
  3. cd review_site
  4. java -jar bin/gerrit.war init --batch --install-all-plugins

5) Run initialization

gerrit init extract the plugins from the upstream war, run their init steps, upgrade the schema and normalize gerrit.config.

For upstream bundled plugins we have forked (such as gitiles), we need to restore our version. That is done by restoring git tracked .jar from our deployment repository. Rest of bundled plugins are untracked in our git repository.

  1. cd plugins
  2. git status
  3. git checkout '*.jar' # restore OUR version of upstream bundled plugins

6) Apply our config Enable and run Puppet:

sudo puppet agent --enable
sudo puppet agent -tv

Take note of any change that might have been applied to gerrit.config: they should be transcribed back to Puppet.

7) Puppet will start up Gerrit for you.

8) Checks

Plugins either on the web https://gerrit.wikimedia.org/r/admin/plugins or via ssh:

  • ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls
  • ssh -p 29418 gerrit-replica.wikimedia.org gerrit plugin ls


sudo journalctl -u gerrit -n 50 --follow
/var/log/gerrit/error_log

Logs are in Kibana https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV

9) Resume monitoring