You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Portal:Data Services/Admin/Runbooks/Create an NFS server
NFS is hosted on physical hardware as well as virtual servers. As of 2022-01-24 we are transitioning most NFS workloads to virtual servers in the cloudinfra-nfs project. The dumps servers (currently labstore1006/1007, soon to be clouddumps1001/1002) will remain on hardware for some time.
Creating a new server relies on a spicerack cookbook. The cookbook is in the wmcs branch and is typically run locally rather than on a cumin host. Details about setting up a local cookbook exec environment can be found at Wikimedia_Cloud_Services_team/EnhancementProposals/Operational_Automation#Local_setup.
Create a Server for a new service
Each volume consists of a persistent cinder volume, a service IP, a service name, and a replaceable VM. The following command will create all of those; the prefix will also be used to create the service name.
$ cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.nfs.add_server --create-storage-volume-size <size in GB> --project cloudinfra-nfs --prefix <name of volume> <name of volume> --flavor <server flavor id> --image <glance image id> --network 7425e328-560c-4f00-8e99-706f3fb90bb4 --service-ip
The newly created server will also run an nfs-exportd service to maintain exports to the new volume. The behavior of that file is configured via the puppet file nfs-mounts.yaml.erb and the results can be found in /etc/exports.d
Create a replacement server for an existing service
To upgrade or replace the VM hosting a given NFS service, first create a detached server. This will contain all the necessary services but will NOT create a service name, a service IP, or a cinder volume. Instead it creates a VM available for failover from an existing server with storage and service name attached:
$ cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.nfs.add_server --project cloudinfra-nfs --prefix <name of volume> <name of volume> --flavor <server flavor id> --image <glance image id> --network 7425e328-560c-4f00-8e99-706f3fb90bb4
Note that the omission of
--create-storage-volume-size prevents creation and attachment of the cinder volume, and the omission of
--service-ip prevents the creation of a new service name or IP.
NFS service failover
For a particular NFS volume, service can be moved from an existing server (likely created using the command in the 'new service' section above) to a passive server <likely created using the 'replacement server' section above) like this:
$ cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.nfs.migrate_service --project cloudinfra-nfs --from-host-id <current server ID> --to-host-id <future server id>
Most clients will handle that change gracefully due to the consistent name and IP. Some clients may seize up or otherwise misbehave if they are in the middle of file activity during the failover.