You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
gitlab-replica.wikimedia.org is a copy of production GitLab. Currently gitlab2001 is the replica of gitlab1001.
The purpose of the GitLab replica is:
- to have a almost identical machine with production Gitlab to test changes
- to verify the installation path developed by S&F for gitlab1001
- to have a host for failover in case production GitLab breaks (WIP)
- to have parity with gerrit-replica
The purpose is not to share the load between both GitLab instances.
Manual sync of replica
To manually sync of production GitLab with the replica the latest backups have to be moved from production GitLab to the replica. You can either use the existing
rsync configuration or cumin. In most cases the most recent backup should bre present under
/srv/gitlab-backup/latest and you can skip this step.
Transfer of backup to replica
For rsync, run the following commands on gitlab1001 to sync the latest backup to gitlab2001:
gitlab1001:~# /usr/bin/rsync -avp --delete /etc/gitlab/config_backup/latest/ rsync://gitlab2001.wikimedia.org/config-backup gitlab1001:~# /usr/bin/rsync -avp --delete /srv/gitlab-backup/latest/ rsync://gitlab2001.wikimedia.org/data-backup
/data-backup are just module names of the
rsyncd config. The backups are copied to
Or use cumin:
cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/etc/gitlab/config_backup/gitlab_config_1628121602_2021_08_05.tar gitlab2001.wikimedia.org:/etc/gitlab/config_backup/ cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/srv/gitlab-backup/1628121868_2021_08_05_13.12.9_gitlab_backup.tar gitlab2001.wikimedia.org:/srv/gitlab-backup/
Restore of backup on replica
The GitLab restore guide should be used. The exception here is that we have to keep the
gitlab.rb from the replica when restoring the config backup.
Before restore, make sure to copy the old
gitlab.rb before following the guide.
sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.bak
Before restore, make sure the backup can be found by GitLab. So we have to move and rename the backups:
cp /srv/gitlab-backup/latest/latest.tar /srv/gitlab-backup/latest_gitlab_backup.tar
After the restore is finished, the settings have to be reapplied to the replica using gitlab-settings:
./settings --instance https://gitlab-replica.wikimedia.org --settings-file settings-replica.yaml update
Automated sync of replica
Automated sync is not fully implemented. The plan is to restore backups of production GitLab every 24 hours.
For autometed sync a cronjob with
rsync is configured on production GitLab and
rsyncd is running on the replica. The job runs every 24h and transfers the most recent data and config backup to the replica.
The most recent backup can be found in:
So make sure to use the timestamp latest when using the
gitlab-backup restore BACKUP=latest command and follow the GitLab restore guide.
The usage of gitlab-replica should be mostly internal for testing, troubleshooting and in case of an emergency. Thus login with SSO needs the groups