You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Platform-specific documentation/HP DL3N0 Gen9
We have a few of these which are used as Labs virtualization servers: labvirt1010 and 1011. These are also used in other systems. Due to being 'Gen 9' systems, they are different from the Gen8 (like the older labvirt nodes) in several ways. Mercifully, the iLO (aka 'mgmt') console still works the same as the DL3N0 boxes, so refer to that page for mgmt commands.
Enter the system setup tool by pressing esc-9 during boot. The terminal emulation of this tool is lousy, so things will scrawl all over your screen and generally be hard to use.
Setting proper power option
* select service options
* Set Processor Power Monitoring and choose disabled
* Press enter, ignore warning message regarding modification by pressing enter again. Select disabled and press enter again.
Embedded user partition
Make sure that this setting is turned off, otherwise partman will hate you:
System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> USB Options -> Embedded User Partition = Disabled
10g nics disabled
Currently we aren't using the 10g network adapter. Disable it in the bios so that the system doesn't try to use it for net boot.
Virtualization is turned off by default. Make this change to enable virtualization:
System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> Virtualization Options -> Virtualization Technology = Enabled
System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> Virtualization Options -> Intel(R) VT-d = Enabled
System Configuration -> BIOS/Platform Configuration (RBSU) -> System Options -> Virtualization Options -> SR-IOV = Enabled
Systems arrive configured with UEFI boot mode, which doesn't work with our existing netboot system. To enable pxe boot, change
System Configuration -> BIOS/Platform Configuration (RBSU) -> Boot Options -> Boot Mode = 'Legacy BIOS Mode'
Each server as 8 1.2 Tb SSD drives. The need to be configured as one big hardware Raid 10 -- modifying the raid config is possible through management console, the text version takes a while to load and will display
no such device on the screen while loading. Once loaded you are dropped into
hpssacli like you would on linux (it is in fact loading linux)
ms-be RAID0 config
An easy way to configure swift backend ms-be machines disks all in raid0 using the console above (order is important )
First, reboot the system and during reboot Press 'ESC+9' to enter for System Utilities. Once in the System Utilities, select System Configuration then Slot 3 : Smart Array P840 Controller. Select Exit and launch HP Smart Storage Administrator(HPSSA). At the next step, an error message will appear 'error: no such device: EMBEDDED250.' there is nothing to do at this point, but wait for the hpssacli prompt (==>)
set target controller slot=3 array all delete forced create type=arrayr0 drivetype=ss_sata create type=arrayr0 drivetype=sata
Mark a disk as failed
It might happen that Linux detects errors while writing to a disk but the raid controller itself doesn't see the disk as failed (e.g. https://phabricator.wikimedia.org/T163690). In these cases it is useful to forcefully mark the physical drive as failed as follows:
set target controller slot=3 pd all show # take note of the disk e.g. 1I:1:5 pd DISK modify disablepd forced
To reenable the LD (not the PD) after the disk has been swapped:
ld NUMBER modify reenable
set target controller slot=3 pd DISK modify led=on
On first install and after the first puppet run there might be messages similar to this showing up on console:
ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20160831/exfield-427) ACPI Error: Method parse/execution failed [\_SB.PMI0._PMM] (Node ffff8a523f04f2f8), AE_AML_BUFFER_LIMIT (20160831/psparse-543) ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20160831/power_meter-338)
This is related to the "power meter" ACPI module loaded, we blacklist the module since https://gerrit.wikimedia.org/r/#/c/356422/ and thus a reboot will make the message disappear.
complete system specs
(Virtualization Server) 2xE5-2697v3 (2.6GHz/14-C), 384GB RAM, 2x10G, P440 HP ProLiant DL360 Gen9 8SFF Configure-to-order Server HP DL360 Gen9 Intel Xeon E5-2697v3 (2.6GHz/14-core/35MB/145W) FIO Processor Kit HP DL360 Gen9 Intel Xeon E5-2697v3 (2.6GHz/14-core/35MB/145W) Processor Kit 12 x HP 32 GB (1x32 GB) Dual Rank x4 DDR4-2133 CAS-15-15-15 Registered Memory Kit HP FLEXFABRIC 10GB 2P 534FLR-SFP+ ADPTR HP Smart Array P440ar/2GB FBWC 12Gb 2-ports Int FIO SAS Controller HP 800W FS Plat Ht Plg Power Supply Kit 8 x Intl DC S3510 1.20 TB 2.5in internal Solid State Drive -SATA - 500 MB/s Maximimum Read Transfer Rate - 440 MB/s Maximum Write Transfer Rate -1 Pack - 256-bit Encryption Standard
RAID controller firmware upgrade
Platform-specific documentation/HP ProLiant DL380 Gen10
The ms-be systems RAID configuration is each disk in its own RAID 0 Starting from the SSD disks first. So the ms-be systems in general comes with a total of 14 disks. Counting from 0 to 13, the ssd's are in slot 12 and 13. You need to create first a RAID 0 for the first SSD disk in slot 12 then another RAID 0 for the SSD in slot 13 so that each SSD's will take as name sda and sdb. After that, do the same for the other 12 disks. At the end you will have:
Array A Array B Array C Array D Array E Array F Array G Array H Array I Array J Array K Array L Array M Array N
Array A being the SSD in slot 12 and Array B the SSD in slot 13
once in BIOS go to "system Configuration" - "Embedded RAID 1 : HPE Smart Array P816i-a SR Gen10 " - "Array Configuration " - "Create Array "
BIOS configuration =
- Disable Internal SD card slot
If this is not disable the installation will failed at disk partition.
Once in BIOS go to "System Configuration" - "System utilities" - "System Options" - "USB Options" - " Internal SD card slot" and change it from enabled to disabled
- Set boot option to "Legacy BIOS mode"
Once in BIOS go to "System Configuration" - "System utilities" - "System Options" -" Boot options" change mode mode from UEFI mode to Legacy BIOS mode