Server Lifecycle

This page describes the lifecycle of Wikimedia servers, starting from the moment we acquire them and until the time we don't own them anymore. A server has various states that it goes through, with several steps that need to happen in each state. The goal is to standardize our processes for 99% of the servers we deploy or decommission and ensure that some necessary steps are taken for consistency, manageability & security reasons.

The inventory tool used is Netbox and each state change for a host is documented throughout this page.

States

Server Lifecycle	Netbox	Racked	Power
requested	none, not yet in Netbox	no	n/a
spare	PLANNED	yes or no	off
staged	STAGED	yes	on
active	ACTIVE	yes	on
failed	FAILED	yes	on or off
decommissioned	INVENTORY	yes	on or off
unracked	OFFLINE	no	n/a
recycled	none, not anymore in Netbox	no	n/a

Server transitions

Diagram of the Server Lifecycle transitions

Requested

• New hardware is requested for use via the instructions on SRE Team requests#Hardware_requests.
• Hardware Allocation Tech will review request, and detail on ticket if we already have a system that meets these requirements, or if one must be ordered.

• If further details are needed, the task will be assigned back to requester for additional information.

• If hardware is already available and request is approved by operations management, system will be allocated, skipping the rest of this process to the Existing System Allocation step.
• If hardware must be ordered, the buyer will gather quotes from our approved vendors & perform initial reviews on quote(s).

• At this time, quotes are still in RT

• Technical review is done by operations team members familiar with hardware in question (see below), attaching their confirmation or corrections.

• If there are corrections, ticket goes back to buyer and requester as needed until issues are clarified otherwise escalate to Systems Architect(s).

• System Architect(s) perform final technical review to ensure the technical correctness, cost effectiveness, & architecture/roadmap alignment; attaching confirmation or corrections.

• If there are corrections, ticket goes back to buyer and requester as needed until issues are clarified otherwise escalate to Operations Management.

• Operations Management reviews ticket and attaches approval(s) or questions as needed & assigns ticket back to buyer.
• Buyer may create an on-site hardware confirmation task, this ticket will confirm all parts, cables, and assorted items are available for incoming system.

• Ticket is assigned to on-site tech, who must confirm or request the required hardware accessories needed to support the system.
• Order may proceed even if all hardware is not on site, depending on missing hardware and lead times.

Existing System Allocation

See the #Decommissioned -> Staged section below.

• Only existing systems (not new) use this step if they are requested.

• If a system must be ordered, please skip this section and proceed to Ordered section.

• The system needs to be either already decommissioned or a reimage is required.
• If all the above are good, the Hardware Allocation Tech will update your Phabricator ticket and the Server Spares page to reflect the allocation.
• Skip following steps until the Installation section.

Ordered

• Only new systems (not existing/reclaimed systems)
• Buyer purchases hardware, attaching ordering details to ticket.
• Once order ships, buyer places inbound shipment ticket with datacenter vendor. Point of Contact details here.
• Buyer assigns RT procurement ticket to the on-site technician to receive in hardware.

Post Order

• An installation/deployment task should be created (if it doesn't already exist) for the overall deployment of the system/OS/service & place in the #operations project.

• You can include the following steps on this ticket for ease of reference (taken from the entirely of the lifecycle document):

 System Deployment Steps:
  [] - mgmt dns entries created/updated (both asset tag & hostname) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - system bios and mgmt setup and tested [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - network switch setup (port description & vlan) [link sub-task for network configuration here, sub-task should include the network project]
  [] - production dns entries created/updated (just hostname, no asset tag entry) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - install_server module updated (dhcp and netboot/partitioning) [done via this task when on-site subtasks complete]
  [] - install OS (note jessie or trusty) [done via this task when network sub-task(s) complete]
  [] - service implementation [done via this task post puppet acceptance]

Requested -> Spare

Receiving Systems On-Site

• Before the new hardware arrives on site, a shipment ticket must be placed to the datacenter to allow it to be received.

• If the shipment has a long enough lead time, the buyer should enter a ticket with the datacenter site. Note sometimes the shipment lead times won't allow this & a shipment notification will instead be sent when shipment arrives. In that event, the on-site technician should enter the receipt ticket with the datacenter vendor.

• New hardware arrives on site & datacenter vendor notifies us of shipment receipt.
• Packing slip for delivery should list an RT # & the RT ticket should have been assigned to the on-site technician for receipt at this time.
• Open boxes, compare box contents to packing slip. Note on slip if correct or incorrect, scan packing slip and attach to ticket.
• Compare packing slip to order receipt in the RT ticket, note results on ticket.
• If any part of the order is incorrect, reply on RT ticket with what is wrong, and assign the ticket to the buyer on the ticket.
• If the entire order was correct, please note on the procurement ticket. Unless the ticket states otherwise, it can be resolved by the receiving on-site technician at that time.
• Assign asset tag to system, enter system into Netbox immediately, even if not in rack location, with:

• Device role (dropdown), Manufacturer (dropdown), Device type (dropdown), Serial Number (OEM Serial number or Service tag), Asset tag, Site (dropdown) State PLANNED, Platform (dropdown), Purchase date, Support expiry date, Procurement ticket (Phabricator or RT).
• Some systems will have a hostname assigned to them at time of order (usually for clustered systems), if it has been assigned, it will be on the procurement ticket, insert that as Name in Netbox. If it has not been assigned, leave the entry without Name for now.

• Hardware warranties should be listed on the order ticket, most servers are three years after ship date.
• Network equipment has one year coverage, which we renew each year as needed for various hardware.

Spare -> Staged

Preparation

• A Phabricator task should exist with racking location and other details; made during the post-order steps above.
• All systems should have the following common bios/ilom settings set: cpu hyperthreading on, cpu virtulization off (except for virt and ganeti hosts), serial redirection to com2, redirection after post off, boot mode to legacy bios, ipmi enabled, confirm boot order to list disk first, set performance options to OS performance per watt (dells).
• Hostname may be assigned (or system may refer to asset tag name until it is allocated for specific role)

• Please see Server naming conventions for details on how hostnames are determined.
• If hostname was not previously assigned, a label with name must be affixed to front and back of server.

• If system has a front LCD, please see instructions on how to set the name on it via Platform-specific documentation

• DNS is updated for the mgmt network connections.

• DNS for mgmt should include both the $assettag.mgmt.site.wmnet as well as $hostname.mgmt.site.wmnet.
• DNS for production network will be set only for hostname, since a system will have a hostname before going on the production network, systems may not have this set if their usage isn't yet determined.

• Netbox entry must be updated to reflect rack location and hostname (if it has been assigned, otherwise keep it empty)
• System Bios & out of band mgmt setttings are configured at this time..

• See the Platform-specific documentation for setup instructions for each system type.
• Serial Redirection and mgmt must be tested at this time

• On-site Tech should fully test the mgmt interface to ensure it responds to ssh, they are able to login, reboot the system, and watch a successful BIOS POST over serial console.

• Switch port(s) are assigned and labeled.

• Label with hostname, if not available label with asset tag.
• VLAN assignment is completed at this time only if system role is known.

• After systems have been racked, if they are not immediately allocated to a service (IE: they are spare), a ticket should be created in core-ops and assigned to the HW Allocation Tech with the asset tags, so they can add to spares list. [@robh can this line be removed?]

Installation setup

• Hostname must be assigned at this point.

• Please see Server naming conventions for details on how hostnames are determined.
• If hostname was not previously assigned, a label with name must be affixed to front and back of server and Netbox updated with the hostname.

• If system has a front LCD, please see instructions on how to set the name on it via Platform-specific documentation.

• DNS setup for production network.

• $assettag.mgmt.$loc.wmnet should have been setup when the system was racked.
• set up $name.mgmt.$loc.wmnet to the same IP as $assettag.mgmt.$loc.wmnet.

• VLAN: Network port is set to proper vlan (and labeled with hostname if not yet labeled.)
  • Do not use the enable keyword (if it's not explicitly disabled, it's enabled)

• Folks who can handle vlan assignments: Chris J, Faidon L, Mark B, Rob H., Arzhel Y.

• Any ops folks who want this ability should speak to our network admins.

• DHCP: Add server to appropriate file in Puppet, based on serial console port and speed:

• modules/install_server/files/dhcpd/linux-host-entries.ttyS0-9600 = com port 1, speed of 9600
• modules/install_server/files/dhcpd/linux-host-entries.ttyS0-115200 = com port 1, speed of 115200
• modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200 = com port 2, speed of 115200 (most hosts)
• You can pull this information from the management of most systems, as described in their specific pages under Platform-specific documentation.

• Decide on partition mapping & add server to modules/install_server/files/autoinstall/netboot.cfg

• Detailed implementation details for our Partman install exist here.
• The majority of systems should use automatic partitioning, which is set by inclusion on the proper line in netboot.cfg.
• Any hardware raid would need to be setup manually via rebooting and entering raid bios.

• Right now there is a mix of hardware and software raid availability.

• File located @ puppet modules/install_server.

• partman recipe used located in modules/install_server
• Please note if you are uncertain on what to pick, you should lean towards LVM.

• Many reasons for this, including ease of expansion in event of filling the disk.

Installation

At this point the host can be installed. From now on the service owner should be able to take over and install the host automatically, asking DC Ops to have a look only if there are issues. As a rule of thumb if the host is part of a larger cluster/batch order, it should install without issues and the service owner should try this path first. If instead the host is the first of a batch of new hardware, than is probably better to ask DC Ops to install the first one. Consider it a new hardware if it differs from the existing hosts by Generation, management card, RAID controller, network cards, BIOS, etc.

Automatic Installation

See the #Reimage section on how to use the reimage script to install a new server. Don' t forget to set the --new CLI parameter.

Change the state in Netbox to STAGED. [TODO: to be added to the reimage script]

Manual installation

• Reboot system and boot from network / PXE boot
• acquires hostname in DNS
• acquires DHCP/autoinstall entries
• gets installed

• Post-Install: get Puppet running
  • Nothing replaces fully understanding how our deployment of Puppet operates, as detailed on the service info page.
  • Warning: if you are rebuilding a pre-existing server (rather than a brand new name), on the puppet master (puppetmaster1001), run puppet cert destroy $server_fqdn to clear out the old certificate before beginning this process. If you already began, also run (on the server you're building, not the puppet master) find /var/lib/puppet/ssl -type f -exec rm {} \; to clean out the client.
  • From puppetmaster1001, sudo /usr/local/sbin/install-console $server_fqdn to log into $server
  • If you're reimaging a WMCS server, run install-console from iron.wikimedia.org
  • On $server, run puppet agent --test

It should whine that it can't get its cert automatically: Exiting; no certificate found and waitforcert is disabled

• on the puppet master (puppetmaster1001), run puppet cert -l to list all pending certificate signings.
• on the puppet master, run puppet cert -s $server_fqdn for the specific server you wish to sign keys for.
• Now again on $server, run puppet agent --enable to administratively enable puppet, and then puppet agent --test. It should now succeed.
• After your first couple of successful puppet runs, you should reboot just to make sure it comes up clean.
• Your host should now appear in puppet stored configs and therefore in Icinga.

• Change the state in Netbox to STAGED.

Spare -> Unracked

When a host in the spare pool has reached its end of life and must be unracked.

• DC Ops perform actions to safely unrack the host, see the #Reclaim to Spares OR Decommission section below.

Staged -> Active

• When a server is placed into service, documentation of the service (not specifically the server) needs to reflect the new server's state. This includes puppet file references, as well as wikitech documentation pages.
  • Example: Some servers have SSH fingerprints listed.
• Service owner pool the host back in production.
• hanges Netbox's to ACTIVE.

Active -> Staged

This transition should be used when reimaging or when a rollback of the STAGED -> ACTIVE transition is needed.

• Service owner perform actions to remove it from production, see the #Remove from production section below.
• Perform the reimage using the available scripts, see the #Reimage section below.
• Service owner changes Netbox's state to STAGED [TODO: include this step into the wmf-auto-reimage script]

Active -> Failed

When a host fails and requires physical maintenance/debugging by DC Ops:

• Service owner perform actions to remove it from production, see the #Remove from production section below.
• service owner changes Netbox's state to FAILED

Active -> Decommissioned

When the host has completed his life in a given role and should decommissioned or returned to the spare pool for re-assignement.

• Service owner perform actions to remove it from production, see the #Remove from production section below.
• Service owner changes Netbox's state to INVENTORY

Failed -> Staged

When the failure has been fixed and the host is ready to be tested before being put back into production.

• Change Netbox's state to STAGED

Failed -> Unracked

When the failure cannot be fixed and the host is not anymore usable.

• DC Ops perform actions to safely unrack the host, see the #Reclaim to Spares OR Decommission section below.

Decommissioned -> Spare

When a decommissioned host is going to be part of the spare pool.

• DC Ops wipe and power down the host, see the #Reclaim to Spares OR Decommission section below.
• DC Ops changes Netbox's state to PLANNED

Decommissioned -> Staged

When a host is decomissioned from one role and immediately returned in service in a different role, usually with a different hostname. (Ideally it should be wiped too)

• Service owner rename the host while reimaging it (see Rename while reimaging below)
• Service owner changes Netbox's state to STAGED

Decommissioned -> Unracked

The host has completed its life and is being unracked

• DC Ops perform actions to safely unrack the host, see the #Reclaim to Spares OR Decommission section below.

Unracked -> Recycled

When the host physically leaves the datacenter.

• DC Ops perform actions to recycle the host, see the #Reclaim to Spares OR Decommission section below.

Server actions

Reimage

The wmf-auto-reimage-host (single host) and wmf-auto-reimage (multiple hosts) scripts allow to automate most of the installation/re-image tasks outlined in this document. They are installed in cumin masters and must be run in a screen/tmux with sudo -i (to load conftool authentication). Read the wmf-auto-reimage -h help page for a full list of options.

Example:

$ sudo -i wmf-auto-reimage  -p T206450 rdb2004.codfw.wmnet

Actions performed by wmf-auto-reimage:

• Updates the Phabricator task
• Validates FQDN of hosts (unless --new or --no-verify are set)
• Downtimes on Icinga (unless --no-downtime is set)
• Depool hosts via conftool (if --conftool is set)
• Sets next boot in PXE mode
• Power cycles or powers on based on current power state
• use the new hostname (if set). Note: It is essential that the new hostname is already set via DHCP and configured in DNS
• Runs puppet once to create the certificate and the signing request to the Puppet maste
• Masks all provided systemd units to prevent them to start automatically during the first Puppet run.
• Triggers the first Puppet run
• Runs Puppet on the Icinga host and set it in dowtime (sometimes this might fail and some alarms may go off)
• Reboots
• Checks if first puppet run is successful
• Umasks the masked systemd units
• Print the conftool commands to re-pool the host (if -c )
• Update the Phabricator task with the result

Remove from production

• A Phabricator ticket should be created detailing the reinstallation in progress.
• System services must be confirmed to be offline. Make sure no other services depend on this server.
• Remove from pybal/LVS (if applicable) - see wmf-auto-reimage option -c/--conftool and consult the LVS page
• Check if server is part of a service group. For example db class machines are in associated db-X.php, memcached in mc.php.
• Remove server entry from DSH node groups (if applicable). For example check operations/puppet:hieradata/common/scap/dsh.yml

Rename while reimaging

This is a hint of a procedure that can be followed to rename a server while doing the reimage. It follow the active -> decommissioned -> staged path.

• Remove the host from production
• Change Netbox's state to INVENTORY
• patch for puppet adjusting install/roles for the new server. Merge it.
• patch DHCP entry, partman entry, Merge it. Run puppet on the install server.
• patch for DNS, rename the production entries, add the new mgmt FQDNs (don't delete old mgmt entries yet). Merge it.
• disable puppet in the server to be reimaged + renamed
• run the wmf-auto-reimage-host script (with the --rename and --rename-mgmt options) on a cumin master (f.e. cumin1001)
• change Netbox state to STAGED
• patch for dns to cleanup DNS entries. Merge it.
• get the physical re-labeling done (open a task for dc-ops)
• update the network port description on switch

Examples of all of this: phab:T199521, phab:T199107, phab:T208824

Reclaim to Spares OR Decommission

TODO: this section should be split in three: Wipe, Unrack and Recycle.

Steps for ANY Opsen

• A Decommission ticket should be created detailing if system is being decommissioned (and removed from datacenter) or reclaimed (wiped of all services/data and set system as spare for reallocation).
  • Please put a full decommission checklist of the steps in the main task description, this ensures none are accidentally missed.
• System services must be confirmed to be offline. Checking everything needed for this step and documenting it on this specific page is not feasible at this time(but we are working to add them all). Please ensure you understand the full service details and what software configuration files must be modified. This document will only list the generic steps required for the majority of servers.
• If server is part of a service pool, ensure it is set to false or removed completely from pybal/LVS.
  • Instructions on how to do so are listed on the LVS page.
• If possible, use tcpdump to verify that no production traffic is hitting the services/ports
• If server is part of a service group, there will be associated files for removal or update. The service in question needs to be understood by tech performing the decommission (to the point they know when they can take things offline.) If assistance is needed, please seek out another operations team member to assist.
  • Example: db class machines are in associated db-X.php, memcached in mc.php.
• Remove server entry from DSH node groups (if any).
  • If the server is part of a service group, common DSH entries are populated from conftool, unless they're proxies or canaries
  • The list of dsh groups is in operations/puppet:hieradata/common/scap/dsh.yaml.
• Remove system entries in site.pp, replace with system entry for role::spare::system, merge changes.
• Remove all hiera data entries for host.
• Run puppet on host to be reclaimed/decommissioned.
  • Leaving the host on, but with role::spare::system will allow it to receive security updates.

Steps for DC-OPS (with network switch access)

• Confirm all puppet manifest entires removal, DSH removal, Hiera data removal.

These steps, once started, must be completed without interruption.

Some of the following steps are covered by the wmf-decommission-host script available on the cluster::management hosts (neodymium/sarin as of Aug. 2018). Those covered by the script are marked as [decom script].

• Disable ALL service level checks in icinga for host.
• Disable puppet on the host (puppet agent --disable)
  • Admin log whenever you disable or enable puppet on a host!
• Remove all references in puppet:
  • remove from site.pp and from hiera data (both individual host files and entries in regex.yaml, if any
  • remove from netboot.cfg (puppet:///modules/install_server/files/autoinstall/netboot.cfg)
  • remove from DHCPD lease file (puppet:///modules/install_server/files/dhcpd/linux-host-entries.ttyS... filename changes based on serial console settings)
  • Instructions on how to do so are on the Puppet service details page.
  • $ puppet node clean <fqdn> [decom script]
  • $ puppet node deactivate <fqdn> [decom script]
    • These 2 commands immediately preceding this should also remove the host from Icinga monitoring:
  • Run puppet on the icinga master (currently einsteinium.wikimedia.org), so that all alerts for the host are removed [not needed if the decom script is run]
  • Alternatively, put the host and all services into downtime for 1+ day, as it will then not alert when the host is powered down & the next puppet run on the icinga host will remove it from monitoring. [decom script]
  • Remove the host from DebMonitor: from one of the cluster::management hosts (neodymium/sarin as of Jul. 2018) run: [decom script]

    sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key
    

• Remove host's port vlan config
  • # show interfaces ge-x/y/z | display inheritance helps identify configuration applied to the port
• disable host's port on switch.
  • If system is being reclaimed for spare, do not change port label.
  • If system is being decommissioned, please do not wipe port description until AFTER it is unracked.
  • THIS MUST BE DONE, or host can be powered up and will be network accessible (but not in puppet and not getting security updates)
  • Move the switch port to interface-range disabled
  • If you can't get puppet to run happily on icinga, get help. If help is not available, stop, renable puppet on the host, and start these steps again once you can get help with icinga.
  • You should either ensure monitoring is removed, or at minimum disable notifications for that host. Don't generate paging alerts for systems you are decommissioning.
• Power down system.

End steps that must be completed without interruption. The following can be done one at a time and/or with long breaks in bewtween.

• Remove DNS entries for the production network.
  • Don't remove the mgmt DNS entries at this time!
    • Reclaims never have mgmt entries removed, and decom servers should keep them until they are wiped and unracked.
• Update associated Phabricator ticket, detailing steps taken and resolution.
  • If system is decommissioned by on-site tech, they can resolve the ticket.
  • If system is reclaimed into spares, ticket should be assigned to the HW Allocation Tech so he can update spares lists for allocation.

Decommission Specific (can be done by DC Ops without network switch access)

• A Phabricator ticket for the decommission of the system should be placed in the #hardware-request project and the appropriate datacenter-specific ops-* project.

• All further decommission steps are handled by the on-site technician.

• Wipe all disks on system with minimum of 3 passes.

• We presently boot off USB version of DBaN.

• Reset all system bios, mgmt bios, & raid bios settings to factory defaults.
• Unrack system
• Change Netbox's state to OFFLINE and reset its rack location.

• Unless another system will be placed in the space vacated immediately, please remove all power & network cables from rack.

• Once server is un-racked, do the following:
• Remove its mgmt DNS entries.
• Remove port description label for decomissioned host's switch port

Network devices specific

• SRX only: ensure autorecovery is disabled (see Juniper doc)
• Wipe the configuration
  • By either running the command request system zeroize media
  • Or Pressing the reset button for 15s
• Confirm the wipe is successful by login to the device via console (root/no password)

Position Assignments

The cycle above references specific position/assignments, without referring to name. To keep the document generic, we'll keep the cycle with positions listed, and just list those folks here.

• Buyer / HW Allocation Tech: Rob H (US), Mark B (EU)
• On-site Tech EQIAD: Chris J
• On-site Tech CODFW: Papaul T
• On-site Tech ULSFO: Rob H
• Director Technical Operations : Mark B
• Operations Technical Review: Mark B, Faidon L

See also

• m:Hardware donation program and traditional blog post announcements
• phabricator:hardware-requests