Portal:Data Services/Admin/Shared storage

Labstore (cloudstore) is the prefix for the naming of a class of servers that fulfill different missions. The common thread is off-compute-host storage use cases that serve the VPS instances and Tools. The majority of labstore clusters provide NFS shares for some purpose.

Clusters

Primary Cluster

Servers: labstore1004, labstore1005

This was previously called the secondary cluster (now it only is in the client mount).

• Tools project share that is used operationally for deployment
• Tools home share that is used for /home in ToolForge
• Misc home and project shares that are used by all NFS enabled projects, except maps

Components: DRBD, NFS, nfs-manage, maintain-dbusers, nfs-exportd, BDSync

Secondary Cluster

Servers: cloudstore1008, cloudstore1009

• An NFS share large enough to be used as general scratch space across projects
  • /data/scratch
• Maps project(s) also have tile generation on a share here temporarily.
• (proposed) Quota limited rsync backup service for Cloud VPS tenants (phab:T209530)
• syncserver is a python service that rsyncs data between the two. The primary server is set via hiera.
• During a failover, it is probable that the maps project servers will need reboots (phab:T224747)

Components: NFS, syncserver, nfs-exportd

Dumps

Servers: labstore1006, labstore1007

• Dumps customer facing storage
  • NFS exports to Cloud VPS projects including Toolforge
  • NFS exports to Analytics servers
  • Rsync origin server for dumps mirroring
  • https://dumps.wikimedia.org (nginx)
• Does NOT use nfs-exportd and nfs, rsync and nginx should remain active on both servers

Components: NFS, nginx, rsync (for mirrors and syncing to stats servers)

Offsite backup

Servers: labstore2003, labstore2004

• labstore2003 acts as a backup server for the "tools-project" logical volume from labstore100[45]
• labstore2004 acts as a backup server for the "misc project" logical volume from labstore100[45]

Components: BDSync, Backups

Components

DRBD

DRBD syncs block devices between two hosts in real time.

Useful commands

DRBD status

cat /proc/drbd

NFS

NFS client operations

When significant changes are made on an NFS server, clients that mount that often need actions taken on them to recover from whatever state they are suddenly in. To this end, the cumin host file backend is there to work in tandem with the nfs-hostlist script. The latter script will generate a list of VMs by project and specified mounts where NFS is mounted. Currently, you must be on a cloudinfra cumin host to run these commands. The current host is cloud-cumin-01.cloudinfra.eqiad1.wikimedia.cloud

The nfs-hostlist script takes several options (some are required):

• -h Show help
• -m <mount> A space-delimited list of "mounts" as defined in the /etc/nfs-mounts.yaml file generated from puppet (it won't accept wrong answers, so this is a pretty safe option)
• --all-mounts Anything NFS mounted (but you can still limit the number of projects)
• -p <project> A space-delimited list of OpenStack projects to run against. This will be further trimmed according to the mounts you selected. (If you used -m maps and -p maps tools, you'll only end up with maps hosts)
• --all-projects Any project mentioned in /etc/nfs-mounts.yaml, but you can still filter by mounts.
• -f <filename> Without this, the script prints to STDOUT.

Example:

1. First, create your host list based on the mounts or projects you know you will be operating on. For example, if I was making a change only to the secondary cluster, which currently serves maps and scratch, one might generate a host list with the command:

   bstorm@cloud-cumin-01:~$ sudo nfs-hostlist -m maps scratch --all-projects -f hostlist.txt
   

   Note that root/sudo is needed because this interacts with cumin's query setup to get hostnames. It will take quite a while to finish because it also calls openstack-browser's API to read Hiera settings.
2. Now you can run a command with cumin across all hosts in hostlist.txt similar to

   bstorm@cloud-cumin-01:~$ sudo cumin --force -x 'F{/home/bstorm/hostlist.txt}' 'puppet agent -t'
   

It is sensible to have the host list generated shortly before the changes will take place to respond quickly as needed with cumin when you need to.

nfs-manage

This script is meant as the entry point to bringing up and taking down the DRBD/NFS stack in its entirety.

nfs-manage status
nfs-manage up
nfs-manage down

nfs-exportd

Dynamically generates the contents of /etc/export.d to mirror active projects and shares as defined in /etc/nfs-mounts.yaml, every 5 minutes.

This daemon fetches project information from OpenStack to know the IPs of the instances and add them to the exports ACL.

See ::labstore::fileserver::exports.

WARNING: there is a known issue, in case some openstack component is misbehaving (for example, keystone), this will typically return a 401. Please don't allow this to make it past the traceback. We want exceptions and failures in the service instead of letting it remove exports. There is also a cron job that backs up exports to /etc/exports.bak.

syncserver

A simple python service that runs an throttled rsync every 5 min after the last one finishes between the scratch and maps from the active server from the secondary cluster to the inactive server.

maintain-dbusers

Debugging info at: Portal:Toolforge/Admin#Regenerate_replica.my.cnf

We maintain the list of accounts to access the Wiki Replicas on the labstore server in the secondary cluster that is actively serving the Tools project share. The script writes out a $HOME/replica.my.cnf file to each user and project home containing MySQL connection credentials. This uses LDAP to get a list of accounts to create.

The credential files are created with the immutable bit set with chattr to prevent deletion by the Tool account.

The code pattern here is that you have a central data store (the db), that is then read/written to by various independent functions. These functions are not 'pure' - they could even be separate scripts. They mutate the DB in some way. They are also supposed to be idempotent - if they have nothing to do, they should not do anything.

Most of these functions should be run in a continuous loop, maintaining mysql accounts for new tool/user accounts as they appear.

populate_new_accounts

• Find list of tools/users (From LDAP) that aren't in the `accounts` table
• Create a replica.my.cnf for each of these tools/users
• Make an entry in the `accounts` table for each of these tools/users
• Make entries in `account_host` for each of these tools/users, marking them as absent

create_accounts

• Look through `account_host` table for accounts that are marked as absent
• Create those accounts, and mark them as present.

If we need to add a new labsdb, we can do so the following way:

• Add it to the config file
• Insert entries into `account_host` for each tool/user with the new host.
• Run `create_accounts`

In normal usage, just a continuous process running `populate_new_accounts` and `create_accounts` in a loop will suffice.

TODO:

 - Support for maintaining per-tool restrictions (number of connections + time)

Known Issues

Some minority portion of users who had legacy accounts in the older Labsdb cluster did not get accounts created when we transitioned from an older labsdb account management system.

We can resolve this by removing the accounts in labsdb1001 and labsdb1003 and regenerating accounts and credentials.

BDSync

We use the WMF bdsync package on both source and destination backup hosts. Backup hosts run a job periodically to sync a remote block device from a remote target to an LVM device locally.

Backups

Uses bdsync via rsync using SSH as a transport to copy block devices over the network.

Mounting a backup

When mounting a backup from a DRBD device, you have to tell the OS what kind of filesystem it is, since it just sees "DRBD".

Example:

$ mount -t ext4 /dev/backup/tools-project /mnt/tools-project/

Additionally, it is very important to unmount the backup as soon as work is done with it, because the backup jobs will fail if the device is mounted.

How to enable NFS for a project

• Add "mount_nfs: false" to Hiera for any VMs that should never mount the NFS volume in the project. It works best at the beginning.
• Find out the GID for the project (getent group project-$name)
• Add it to modules/labstore/temmplates/nfs-mounts.yaml.erb (example)
• Run sudo puppet agent -tv on labstore1004
• Create the shared folders on labstore1004 as /srv/misc/shared/<project_name>/home and /srv/misc/shared/<project_name>/project as appropriate. Leave ownership with "root:root". That is normal.
• Run puppet on the project VMs.