You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Difference between revisions of "GitLab/Replica"

From Wikitech-static
Jump to navigation Jump to search
imported>Jelto
 
imported>Jelto
Line 11: Line 11:


=== Manual sync of replica ===
=== Manual sync of replica ===
to manually sync of production GitLab with the replica the latest backups have to be moved from production GitLab to the replica:<syntaxhighlight lang="bash">
to manually sync of production GitLab with the replica the latest backups have to be moved from production GitLab to the replica. You can either use the existing <code>rsync</code> configuration or cumin.
 
For rsync, run the following commands on gitlab1001 to sync the latest backup to gitlab2001:<syntaxhighlight lang="bash">
gitlab1001:~# /usr/bin/rsync -avp --delete /etc/gitlab/config_backup/latest/ rsync://gitlab2001.wikimedia.org/config-backup
gitlab1001:~# /usr/bin/rsync -avp --delete /srv/gitlab-backup/latest/ rsync://gitlab2001.wikimedia.org/data-backup
</syntaxhighlight><code>/config-backup</code> and <code>/data-backup</code> are just module names of the <code>rsyncd</code> config. The backups are copied to <code>/etc/gitlab/config_backup/latest</code> and <code>/srv/gitlab-backup/latest</code>.
 
Or use cumin:<syntaxhighlight lang="bash">
cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/etc/gitlab/config_backup/gitlab_config_1628121602_2021_08_05.tar gitlab2001.wikimedia.org:/etc/gitlab/config_backup/
cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/etc/gitlab/config_backup/gitlab_config_1628121602_2021_08_05.tar gitlab2001.wikimedia.org:/etc/gitlab/config_backup/
cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/srv/gitlab-backup/1628121868_2021_08_05_13.12.9_gitlab_backup.tar gitlab2001.wikimedia.org:/srv/gitlab-backup/
cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/srv/gitlab-backup/1628121868_2021_08_05_13.12.9_gitlab_backup.tar gitlab2001.wikimedia.org:/srv/gitlab-backup/
Line 21: Line 28:


=== Automated sync of replica ===
=== Automated sync of replica ===
Automated sync is not implemented currently. The plan is to restore backups of production GitLab every 24 hours.
Automated sync is not fully implemented. The plan is to restore backups of production GitLab every 24 hours.
 
For autometed sync a cronjob with <code>rsync</code> is configured on production GitLab and <code>rsyncd</code> is running on the replica. The job runs every 24h and transfers the most recent data and config backup to the replica.<ref>https://gerrit.wikimedia.org/r/c/operations/puppet/+/710948</ref>


=== Usage ===
=== Usage ===
The usage of gitlab-replica should be mostly internal for testing, troubleshooting and in case of an emergency. Thus login with SSO needs the groups <code>ops</code>, <code>wmf</code> or <code>nda</code>.
The usage of gitlab-replica should be mostly internal for testing, troubleshooting and in case of an emergency. Thus login with SSO needs the groups <code>ops</code>, <code>wmf</code> or <code>nda</code>.

Revision as of 13:36, 11 August 2021

gitlab-replica.wikimedia.org is a copy of production GitLab. Currently gitlab2001 is the replica of gitlab1001.

The purpose of the GitLab replica is:

  • to have a almost identical machine with production Gitlab to test changes
  • to verify the installation path developed by S&F for gitlab1001
  • to have a host for failover in case production GitLab breaks (WIP)
  • to have parity with gerrit-replica

The purpose is not to share the load between both GitLab instances.

Manual sync of replica

to manually sync of production GitLab with the replica the latest backups have to be moved from production GitLab to the replica. You can either use the existing rsync configuration or cumin.

For rsync, run the following commands on gitlab1001 to sync the latest backup to gitlab2001:

gitlab1001:~# /usr/bin/rsync -avp --delete /etc/gitlab/config_backup/latest/ rsync://gitlab2001.wikimedia.org/config-backup
gitlab1001:~# /usr/bin/rsync -avp --delete /srv/gitlab-backup/latest/ rsync://gitlab2001.wikimedia.org/data-backup

/config-backup and /data-backup are just module names of the rsyncd config. The backups are copied to /etc/gitlab/config_backup/latest and /srv/gitlab-backup/latest. Or use cumin:

cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/etc/gitlab/config_backup/gitlab_config_1628121602_2021_08_05.tar gitlab2001.wikimedia.org:/etc/gitlab/config_backup/
cumin1001:~$ sudo SSH_AUTH_SOCK=/run/keyholder/proxy.sock scp -3 gitlab1001.wikimedia.org:/srv/gitlab-backup/1628121868_2021_08_05_13.12.9_gitlab_backup.tar gitlab2001.wikimedia.org:/srv/gitlab-backup/

When the backup archives are transferred to the replica the GitLab restore guide should be used. The exception here is that we have to keep the gitlab.rb from the replica when restoring the config backup. So make sure to copy the old gitlab.rb before following the guide.

sudo cp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb.bak

When the restore is finished, the settings have to be reapplied to the replica using gitlab-settings:

./settings --instance https://gitlab-replica.wikimedia.org update

Automated sync of replica

Automated sync is not fully implemented. The plan is to restore backups of production GitLab every 24 hours.

For autometed sync a cronjob with rsync is configured on production GitLab and rsyncd is running on the replica. The job runs every 24h and transfers the most recent data and config backup to the replica.[1]

Usage

The usage of gitlab-replica should be mostly internal for testing, troubleshooting and in case of an emergency. Thus login with SSO needs the groups ops, wmf or nda.