You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(wm-bot: Testing - cookbook ran by dcaro@vulcanus)
imported>Stashbot
(wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169)
(177 intermediate revisions by the same user not shown)
Line 1: Line 1:
=== 2022-09-27 ===
* 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@wmf3169
* 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
=== 2022-09-26 ===
* 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:31 andrewbogott: rebooting cloudvirt1028 for [[phab:T317391|T317391]]
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-{{Gerrit|eb0413dfb9d0}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-{{Gerrit|d2cdd8abb2bf}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-{{Gerrit|06b6f53fd899}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-{{Gerrit|36bc7f79b4d5}}, use this to unset). - cookbook ran by andrew@buster
* 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-{{Gerrit|35e675abce32}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-{{Gerrit|66f54ba9801c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-{{Gerrit|58aaa9c8e021}}, use this to unset). - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-{{Gerrit|23c636d6554e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-{{Gerrit|1aa1cfe4c5af}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-{{Gerrit|077ffa1c1c5e}}, use this to unset). - cookbook ran by andrew@buster
* 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-{{Gerrit|ef7785a4af4a}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-{{Gerrit|1e768b602a1b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-{{Gerrit|074d82582cb4}}, use this to unset). - cookbook ran by andrew@buster
* 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-{{Gerrit|cd1cf910f8e1}}, use this to unset). - cookbook ran by andrew@buster
* 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-{{Gerrit|29b7699b692f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-{{Gerrit|5c41992192b4}}, use this to unset). - cookbook ran by andrew@buster
* 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-{{Gerrit|6ad4dcb2a105}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-{{Gerrit|8a6d481dea7d}}, use this to unset). - cookbook ran by andrew@buster
* 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-{{Gerrit|8f43c253e499}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
* 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
* 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin ([[phab:T318047|T318047]])
* 10:14 arturo: deployed new version of maintain-dbusers ([[phab:T318047|T318047]])
=== 2022-09-25 ===
* 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-{{Gerrit|3993c1e8b593}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-{{Gerrit|8663db77ce66}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-{{Gerrit|e5625d209b57}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-{{Gerrit|b3a64bd6b18d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-{{Gerrit|afd57ae1ca55}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
=== 2022-09-24 ===
* 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
* 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022
=== 2022-09-22 ===
* 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-{{Gerrit|538741a2f801}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-{{Gerrit|c41d1e1c8c8b}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-{{Gerrit|6efde61db3a3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-{{Gerrit|ff23055a0527}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-{{Gerrit|fd671c398642}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-{{Gerrit|bc66789d37c8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-{{Gerrit|cce5edaa1f08}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-{{Gerrit|eeee37007d14}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-{{Gerrit|76d499eccfa1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-{{Gerrit|677853fb9ef7}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-{{Gerrit|66df673c7ed8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-{{Gerrit|2ed2e474c89f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-{{Gerrit|265f6386f396}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-{{Gerrit|90ab1f492aa8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-{{Gerrit|c571b3df2945}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-{{Gerrit|513767f95b1d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-{{Gerrit|b0005c8164d1}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-{{Gerrit|0d513ad7acb5}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-{{Gerrit|2f49d1033f78}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-{{Gerrit|341981b3887f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-{{Gerrit|139b4d119ef8}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-{{Gerrit|47d87262d0ed}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
=== 2022-09-20 ===
* 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-{{Gerrit|4f52f61d503f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-{{Gerrit|7e9d518be217}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-{{Gerrit|2b18032bbf3c}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-{{Gerrit|ee6bbbf029fa}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-{{Gerrit|a356c39f3330}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-{{Gerrit|01795b1e62ce}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-{{Gerrit|5a7dc56060b6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-{{Gerrit|303d16801e28}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-{{Gerrit|e20cbb877755}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-{{Gerrit|514ca5cdc76d}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 8f2d2d32-e99e-4e2c-9472-{{Gerrit|ca33b577d01f}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:00 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 66ee7d87-cee1-4c8f-b5c3-{{Gerrit|5f58a6cea336}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:59 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 69035b38-9910-46f1-9582-{{Gerrit|3105c1ba4d16}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:52 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 5a94c9d1-c956-4b03-a92b-{{Gerrit|b112810906a4}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:51 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:50 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 216a9120-4b9a-4be3-99db-{{Gerrit|3fd5fdc25146}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:49 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 46fb1964-3cf4-47fd-ad79-{{Gerrit|475f33c8ed38}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:48 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b04a4f8c-db82-4258-8608-{{Gerrit|2a78ec9193ed}}, use this to unset). - cookbook ran by andrew@buster
* 19:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 7ae7c258-ecb8-47d1-a991-{{Gerrit|3fa617e305bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:45 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 377fb0e5-132e-4e7a-9079-{{Gerrit|37937c3d42bf}}, use this to unset). - cookbook ran by andrew@buster
* 19:44 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 6c4222ae-796e-4890-8820-{{Gerrit|9e7ce5438f2a}}, use this to unset). - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:43 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:38 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:34 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:34 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:31 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: eff98e2e-ae99-41d0-a68e-{{Gerrit|671d941fcf21}}, use this to unset). - cookbook ran by andrew@buster
* 19:30 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5310f1eb-0425-4f59-8fb5-{{Gerrit|1d9c6f6d2f23}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:29 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:25 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: d8ab4682-026a-4ddc-bfc5-{{Gerrit|48166bd9789a}}, use this to unset). - cookbook ran by andrew@buster
* 19:24 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:23 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:23 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: eb2f9d94-8ea5-48d5-a336-{{Gerrit|97dd407b1ce3}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:19 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:18 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 305fe2d7-913e-4638-95c2-{{Gerrit|4e810dc6b2a3}}, use this to unset). - cookbook ran by andrew@buster
* 19:14 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5bb3a66b-9450-4230-8d09-{{Gerrit|d78473d72ba2}}, use this to unset). - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:13 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:07 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: b1e39b49-3a88-4e42-913c-{{Gerrit|9892eafad447}}, use this to unset). - cookbook ran by andrew@buster
* 19:06 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 andrewbogott: putting cloudvirt1049-1052 into 'ceph' pool, taking out of 'spare' pool. cloudvirt1053 will remain our only spare.
* 19:02 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:02 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:00 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:59 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Drained 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 8f24ea75-e107-4a13-8bd6-{{Gerrit|950b941b8e03}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:58 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:57 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:56 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 5c3e8fa4-cd1e-43f4-a423-{{Gerrit|ba7fe2459d4e}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: e3fe22d6-932c-4502-8838-{{Gerrit|82cdc4c8f1c6}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:55 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:54 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Safe reboot of 'cloudvirt1053.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:53 wm-bot2: Unset cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: fd824b92-b6ff-4631-a2d4-{{Gerrit|debb25d48223}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:52 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:51 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 52d5cd5b-0f56-453a-85f8-{{Gerrit|0b5e656ae7f9}}, use this to unset). - cookbook ran by andrew@buster
* 18:49 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:49 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: b92f66a3-a487-4515-b68b-{{Gerrit|1c257a57b893}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
=== 2022-09-19 ===
* 20:07 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:07 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 4bfca6cd-dfbc-4a79-9203-{{Gerrit|b432bfeee5c2}}, use this to unset). ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. ([[phab:T317391|T317391]]) - cookbook ran by andrew@buster
* 19:41 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:21 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 70479325-609b-4349-9094-{{Gerrit|739eb9b3bb30}}, use this to unset). - cookbook ran by andrew@buster
* 19:21 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
=== 2022-09-14 ===
* 16:57 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 16:53 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 11:01 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:57 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:09 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 10:06 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 09:46 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:43 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-09-13 ===
* 12:16 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:16 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:11 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
* 12:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:36 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-09-10 ===
* 15:37 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)
=== 2022-09-08 ===
* 19:12 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud
=== 2022-09-07 ===
* 10:18 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:14 wm-bot2: Finished rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Rebooting node cloudcephosd1032.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:10 wm-bot2: Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:41 dhinus: Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)
=== 2022-08-30 ===
* 14:59 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for [[phab:T296561|T296561]]
* 10:43 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:39 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:38 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:39 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:14 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:11 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-08-25 ===
* 15:14 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:14 wm-bot2: Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 15:02 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:59 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 14:58 wm-bot2: Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-08-24 ===
* 22:07 andrewbogott: replaced cloudservices1003 with cloudservices1005 [[phab:T304888|T304888]]
* 10:45 wm-bot2: Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:45 wm-bot2: Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:37 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:34 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 10:33 wm-bot2: Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-08-23 ===
* 13:46 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:46 wm-bot2: Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:27 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:24 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:22 wm-bot2: Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-08-21 ===
* 21:12 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart
=== 2022-08-20 ===
* 07:39 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover ([[phab:T315718|T315718]])
* 07:23 dcaro_away: cloudvirt1023 seems to have gotten some hardware issue from racadm lclog view "System CPU Resetting.", rebooting and doing memory checks ([[phab:T315718|T315718]])
=== 2022-08-19 ===
* 17:06 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes [[phab:T310795|T310795]]
=== 2022-08-18 ===
* 13:25 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:25 wm-bot2: Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:15 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:12 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 13:10 wm-bot2: Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 07:29 dcaro: Starting up all the osd daemons on cloudcephosd1025 ([[phab:T314870|T314870]])
=== 2022-08-17 ===
* 10:50 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:49 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:50 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:49 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@foz
* 09:16 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
* 09:16 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@Francesco’s-MacBook-Pro
=== 2022-08-16 ===
* 22:39 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate
* 17:41 andrewbogott: removing cloudvirt1025 from the 'ceph' aggregate and adding it to the 'maintenance' aggregate
* 17:40 andrewbogott: reimaging cloudvirt1025 after I accidentally deleted the hw raid
* 17:38 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1003@rbd#RBD --newhost cloudcontrol1005@rbd#RBD
* 17:37 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1004@rbd#RBD --newhost cloudcontrol1006@rbd#RBD
* 16:26 wm-bot2: Ceph cluster at eqiad1 set out of maintenance. - cookbook ran by dcaro@vulcanus
* 15:43 wm-bot2: Restarting the osd daemons from nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,c
* 15:42 wm-bot2: Finished restarting all the OSD daemons from the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 15:38 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:07 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:02 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 12:59 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
=== 2022-08-14 ===
* 18:36 taavi: deleted the http keystone endpoints from the keystone service catalog
=== 2022-08-11 ===
* 13:57 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
* 08:42 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:42 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 08:36 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
=== 2022-08-10 ===
* 13:10 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
* 13:06 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster ([[phab:T314870|T314870]]) - cookbook ran by fran@MacBook-Pro.station
=== 2022-08-04 ===
* 17:16 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling
* 16:32 taavi: restart neutron-l3-agent to pick up rabbit config changes
* 15:12 andrewbogott: stopping rabbitmq on cloudcontrol1xxx
* 09:57 taavi: stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom
=== 2022-08-03 ===
* 20:55 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service
* 20:41 andrewbogott: restarting  neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq
=== 2022-08-02 ===
* 14:07 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html
* 13:54 andrewbogott: shutting down basically all of codfw1dev to support pdu maintenance -- all the ceph OSDs will lose power so best to have everything stopped.
=== 2022-07-27 ===
* 19:32 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54
* 16:33 andrewbogott: here is a test message in the admin channel
=== 2022-07-25 ===
* 13:43 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]
=== 2022-07-22 ===
* 16:41 taavi: depool cloudweb1003/1004 since horizon seems to be having issues
* 16:22 taavi: pooling cloudweb1003/1004 now that grant issues are sorted
=== 2022-07-21 ===
* 18:26 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes
* 18:06 andrewbogott: pooling cloudweb1003 and 1004 for wikitech, horizon, striker
=== 2022-07-20 ===
* 18:02 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 ([[phab:T313400|T313400]])
* 17:45 bd808: `sudo service striker restart` on labweb1002
* 17:43 bd808: `sudo service striker restart` on labweb1001
* 17:10 dcaro: things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 ([[phab:T313400|T313400]])
* 16:26 dcaro: things seem stable, trying to bring up a third, cloudcontrol1005 ([[phab:T313400|T313400]])
* 15:51 dcaro: things seem stable now with one rabbit node, trying to bring up a second ([[phab:T313400|T313400]])
* 14:16 dcaro: stopping rabbin on cloudcontrol1004, leaving only 1003 alive ([[phab:T313400|T313400]])
* 13:17 dcaro: restarting the whole rabbit cluster ([[phab:T313400|T313400]])
=== 2022-07-19 ===
* 16:30 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:30 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:26 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:18 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 16:14 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 16:01 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:57 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-{{Gerrit|2b010960ed6a}}, use this to unset). - cookbook ran by andrew@buster
* 15:56 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:49 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-{{Gerrit|363a7dea8d69}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-{{Gerrit|911b14f2314d}}, use this to unset). - cookbook ran by andrew@buster
* 15:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-{{Gerrit|0d4638acc9c3}}, use this to unset). - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:46 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:45 andrewbogott: adding new hosts to the 'ceph' aggregate: cloudvirt1046, 1047, 1048
* 15:44 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:44 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:42 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:40 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:38 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:37 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:33 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 32d9fdb7-6c42-4cf4-9950-{{Gerrit|6e2442255161}}, use this to unset). - cookbook ran by andrew@buster
* 15:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:16 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 56ce1388-792c-442c-bb89-{{Gerrit|e4c3869b0acf}}, use this to unset). - cookbook ran by andrew@buster
* 15:15 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:15 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3f84bf2e-24de-4ebe-8117-{{Gerrit|0f4a5de29d09}}, use this to unset). - cookbook ran by andrew@buster
* 15:14 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:14 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:13 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 5418e923-997c-49ae-b937-{{Gerrit|35fc0c3038eb}}, use this to unset). - cookbook ran by andrew@buster
* 15:12 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: c3dfdc2b-0725-4d01-87f2-{{Gerrit|440a5ac4694c}}, use this to unset). - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:11 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:09 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 15:04 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 15:00 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:54 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:50 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 49dbf7de-c58a-4b68-bc7b-{{Gerrit|f001a047f4c7}}, use this to unset). - cookbook ran by andrew@buster
* 14:46 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:46 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: a4e95168-39b5-452d-8fce-{{Gerrit|7875ae44a62b}}, use this to unset). - cookbook ran by andrew@buster
* 14:44 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:44 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: b46234ee-9900-4708-a815-{{Gerrit|4324379be48c}}, use this to unset). - cookbook ran by andrew@buster
* 14:43 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:43 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:42 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 14:39 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:38 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 12bc87a9-7602-4795-818c-{{Gerrit|f7151b1c9ead}}, use this to unset). - cookbook ran by andrew@buster
* 14:37 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:37 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:35 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: ba787a26-62db-4662-ade2-{{Gerrit|2e3061b7390d}}, use this to unset). - cookbook ran by andrew@buster
* 14:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 6f0e79c6-2d91-478e-92df-{{Gerrit|0f65de48212f}}, use this to unset). - cookbook ran by andrew@buster
* 14:29 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 7f577c0f-0182-498f-b198-{{Gerrit|307d51df8c3a}}, use this to unset). - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: bd1bff92-bfb8-483d-a719-{{Gerrit|13fbdf3d8b21}}, use this to unset). - cookbook ran by andrew@buster
* 14:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: c9cd7dbb-c5ad-4364-a8e0-{{Gerrit|afb52ef98683}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4deca8a4-f73f-4b42-b4be-{{Gerrit|acfe61421ed0}}, use this to unset). - cookbook ran by andrew@buster
* 14:18 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 6d0d58b2-8450-480d-94f2-{{Gerrit|93a257f25575}}, use this to unset). - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:17 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 14:13 dcaro: deleting all the leftover fullstack images (was due to max number of mysql connections reached)
* 04:51 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:47 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:46 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 432747a2-9a15-44d4-ba8a-{{Gerrit|4e7e87eb8b76}}, use this to unset). - cookbook ran by andrew@buster
* 04:45 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:45 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1c5adb8c-efcf-4036-bf22-{{Gerrit|0eef364ae399}}, use this to unset). - cookbook ran by andrew@buster
* 04:44 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:41 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 26feb897-9eb0-40b6-9525-{{Gerrit|4cf44f81638d}}, use this to unset). - cookbook ran by andrew@buster
* 04:41 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:40 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: cd2424d3-c9c9-4837-93b1-{{Gerrit|a6d8271e7873}}, use this to unset). - cookbook ran by andrew@buster
* 04:39 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:39 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1e85ec01-81c4-4664-b14f-{{Gerrit|b1cf6417988c}}, use this to unset). - cookbook ran by andrew@buster
* 04:38 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f498aaa4-fad3-42f8-92f4-{{Gerrit|48a98c108456}}, use this to unset). - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:37 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 0ed281f3-d9b2-4c0a-bf88-{{Gerrit|db42fe848b2f}}, use this to unset). - cookbook ran by andrew@buster
* 04:35 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f417dffc-fa1b-4e65-938a-{{Gerrit|49e54b6fac6d}}, use this to unset). - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:34 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:33 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 923926e0-7194-4e13-98c4-{{Gerrit|a4f703b2907b}}, use this to unset). - cookbook ran by andrew@buster
* 04:32 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:32 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: ca8f729b-92ad-45ec-b5da-{{Gerrit|55987b0fc9c2}}, use this to unset). - cookbook ran by andrew@buster
* 04:29 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:29 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: abe6992d-dfba-4e50-993d-{{Gerrit|de0a551a177a}}, use this to unset). - cookbook ran by andrew@buster
* 04:24 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: df91107d-bf34-49db-9756-{{Gerrit|44ad06162683}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f45ec0b1-83c9-49e0-a3a8-{{Gerrit|897d5a48414f}}, use this to unset). - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 504ae503-bfad-41bd-9079-{{Gerrit|fb53a9c82a62}}, use this to unset). - cookbook ran by andrew@buster
* 04:17 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:17 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 8410faf9-667d-443d-bd17-{{Gerrit|bb3ca8e7f725}}, use this to unset). - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:11 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 40441a37-0dd7-44a8-960b-{{Gerrit|f74f98f53618}}, use this to unset). - cookbook ran by andrew@buster
* 04:10 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:10 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 517cff2b-a643-4714-a31f-{{Gerrit|6bbe0767656a}}, use this to unset). - cookbook ran by andrew@buster
* 04:08 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 50b63ad6-4d8b-4fcf-9c17-{{Gerrit|27a106b6ec52}}, use this to unset). - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:07 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: d912bdf5-54c7-490a-bdbe-{{Gerrit|ba9a23f07f4b}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: e75f16d0-e42f-4d72-a3bd-{{Gerrit|be0cbc5f200a}}, use this to unset). - cookbook ran by andrew@buster
* 04:06 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 87f6c417-3703-4f73-9876-{{Gerrit|20f6767dc8e1}}, use this to unset). - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:05 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: be14f869-fdb0-4c38-84ff-{{Gerrit|3d1e00d227eb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: feb93ea7-2c64-4831-b140-{{Gerrit|dad8311dbcef}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: c65a23e3-a809-4a75-88e4-{{Gerrit|29a8b351ecdb}}, use this to unset). - cookbook ran by andrew@buster
* 04:02 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:01 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:00 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:57 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:56 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8c8f64ca-a4e8-47cf-a2ce-{{Gerrit|23059109fb6a}}, use this to unset). - cookbook ran by andrew@buster
* 03:55 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:55 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:54 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 9d0457e8-a861-46ce-ac35-{{Gerrit|53975a46018e}}, use this to unset). - cookbook ran by andrew@buster
* 03:53 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:53 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:40 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:38 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:30 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 851d3b34-68f7-4c57-920b-{{Gerrit|2a64a9ea573d}}, use this to unset). - cookbook ran by andrew@buster
* 03:29 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:29 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 1060ab82-d53a-4d48-8e15-{{Gerrit|3198cabcfc2f}}, use this to unset). - cookbook ran by andrew@buster
* 03:17 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:17 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: dde1dc09-44b5-42af-ac49-{{Gerrit|c58dbbae7cb4}}, use this to unset). - cookbook ran by andrew@buster
* 03:14 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:14 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: b59fa99b-8713-4e63-8c56-{{Gerrit|08bd71fdfbe3}}, use this to unset). - cookbook ran by andrew@buster
* 03:13 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:13 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 3547c843-6b15-4151-b452-{{Gerrit|6f618a886b7b}}, use this to unset). - cookbook ran by andrew@buster
* 03:07 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 6f5808d0-afb2-4222-ae8e-{{Gerrit|4d953d49cd63}}, use this to unset). - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:06 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: da738ba3-883f-4d76-bb33-{{Gerrit|7eb551de5fc2}}, use this to unset). - cookbook ran by andrew@buster
* 03:05 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:05 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:04 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:03 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f90ca4c1-78af-46bd-9262-{{Gerrit|0fa4edc2898e}}, use this to unset). - cookbook ran by andrew@buster
* 03:02 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:02 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:00 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 62383663-9ae6-4a15-a2a6-{{Gerrit|62be06adbc96}}, use this to unset). - cookbook ran by andrew@buster
* 02:59 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:59 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 6e0da344-49ad-451f-8b8d-{{Gerrit|ea8f853a6f89}}, use this to unset). - cookbook ran by andrew@buster
* 02:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:52 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: c9ec5b7c-aa9f-4afd-a417-{{Gerrit|54c52c07ca47}}, use this to unset). - cookbook ran by andrew@buster
* 02:51 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:51 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 5509c648-58f6-49fd-90e8-{{Gerrit|ac9aa66e8b04}}, use this to unset). - cookbook ran by andrew@buster
* 02:48 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:48 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:46 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: af4ba9ad-e765-4087-86aa-{{Gerrit|9111c0821175}}, use this to unset). - cookbook ran by andrew@buster
* 02:46 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:45 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: 6da98123-8838-4d2e-8659-{{Gerrit|d769bfa292fe}}, use this to unset). - cookbook ran by andrew@buster
* 02:44 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:44 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 02:43 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:42 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 91860079-8fff-40db-9de7-{{Gerrit|17741d528ad6}}, use this to unset). - cookbook ran by andrew@buster
* 02:41 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:41 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:39 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: a6a0f414-d305-4a09-87d2-{{Gerrit|066eec83642b}}, use this to unset). - cookbook ran by andrew@buster
* 02:39 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:38 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 6000898c-4e5e-49c8-8841-{{Gerrit|3262d6da0366}}, use this to unset). - cookbook ran by andrew@buster
* 02:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 24a8b50e-5c04-40e6-a732-{{Gerrit|f0232b83e240}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8d259aca-1803-4346-b5a4-{{Gerrit|a447673b3537}}, use this to unset). - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:33 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 704aca7f-0069-4a4a-b8de-{{Gerrit|1aba7b9ca7ef}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04c0fba3-e2cb-40b4-abfd-{{Gerrit|d24165c9bd55}}, use this to unset). - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 25e5e0c8-5c27-460e-9b44-{{Gerrit|1cbd1c2fb551}}, use this to unset). - cookbook ran by andrew@buster
* 02:28 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5e62968a-0bb6-43b1-ae5e-{{Gerrit|e352442ef976}}, use this to unset). - cookbook ran by andrew@buster
* 02:24 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:24 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:21 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5b1a8e8f-8699-433d-b764-{{Gerrit|197862dc5cf1}}, use this to unset). - cookbook ran by andrew@buster
* 02:20 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:20 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:15 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 299c466e-602b-4a8e-bb37-{{Gerrit|156553dbe426}}, use this to unset). - cookbook ran by andrew@buster
* 02:13 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:13 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:12 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6a64d619-c484-411f-ad9d-{{Gerrit|89d43b063737}}, use this to unset). - cookbook ran by andrew@buster
* 02:02 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:02 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-{{Gerrit|bb4aa0a75e45}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: b2f8c6a2-7224-4fb1-8e12-{{Gerrit|dff6dda02380}}, use this to unset). - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:46 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:40 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: c0f62e0e-7865-4e77-a238-{{Gerrit|d707ceed53a8}}, use this to unset). - cookbook ran by andrew@buster
* 00:38 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:38 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 96a368d5-232f-4187-b0bb-{{Gerrit|5923a50fff71}}, use this to unset). - cookbook ran by andrew@buster
* 00:36 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:36 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 31b6bfcb-81cb-4d89-b977-{{Gerrit|190899ea2e64}}, use this to unset). - cookbook ran by andrew@buster
* 00:35 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:35 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 50f763d4-36ea-4241-8e71-{{Gerrit|cc6aa20ccc9e}}, use this to unset). - cookbook ran by andrew@buster
* 00:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: cd3d17af-ee15-4a9f-8750-{{Gerrit|66b97a46ea12}}, use this to unset). - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:27 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:22 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 9b075166-d424-42c0-8e06-{{Gerrit|f74a47abb798}}, use this to unset). - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
=== 2022-07-18 ===
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: e0aab64f-d911-4d7e-9a97-{{Gerrit|c59d9868ceea}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 447d9c7a-a00e-41db-93ed-{{Gerrit|6d2bfc66e5df}}, use this to unset). - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:11 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 46c2ddf2-50c3-4824-ba72-{{Gerrit|d7fb3fa4c5e0}}, use this to unset). - cookbook ran by andrew@buster
* 23:01 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6facdf4f-7bb8-42cd-8ac8-{{Gerrit|b4412fc4cf70}}, use this to unset). - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:00 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:52 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 23190573-fc0d-4872-8343-{{Gerrit|e35b84132028}}, use this to unset). - cookbook ran by andrew@buster
* 22:51 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:51 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 990313c4-5fa6-4de7-a8bf-{{Gerrit|b361f0c79e90}}, use this to unset). - cookbook ran by andrew@buster
* 22:50 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:50 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:48 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 59367d39-bd2f-47fd-becd-{{Gerrit|255bf823ff12}}, use this to unset). - cookbook ran by andrew@buster
* 22:42 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:42 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:40 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 38602815-d187-4455-9676-{{Gerrit|59607505bba7}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 00f21e1b-d014-4bc5-995f-{{Gerrit|e25201fc77c4}}, use this to unset). - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:39 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: d05d07b2-8c60-46de-a89e-{{Gerrit|76987901901b}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 47f6b9e6-ceab-4fb6-8f76-{{Gerrit|0de4341d0841}}, use this to unset). - cookbook ran by andrew@buster
* 22:31 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:31 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:30 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 599042a5-da25-4985-b502-{{Gerrit|328fe40a72d5}}, use this to unset). - cookbook ran by andrew@buster
* 22:29 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:29 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: a554d107-c14d-41f8-b180-{{Gerrit|093e2cf73e72}}, use this to unset). - cookbook ran by andrew@buster
* 22:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9fef70c7-6f7d-4b33-a9bc-{{Gerrit|8c39630dc182}}, use this to unset). - cookbook ran by andrew@buster
* 22:26 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 3744c0fa-084d-4198-98ca-{{Gerrit|66c7a5210f41}}, use this to unset). - cookbook ran by andrew@buster
* 22:02 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 01caf1e1-df64-474a-94d5-{{Gerrit|f37751e23d62}}, use this to unset). - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:01 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 3018c2fc-9cd5-45c8-a160-{{Gerrit|d5ad5728c8d5}}, use this to unset). - cookbook ran by andrew@buster
* 21:59 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:59 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: d760f916-32ea-4194-8974-{{Gerrit|f36064a9866c}}, use this to unset). - cookbook ran by andrew@buster
* 21:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: cb2a38f3-e9ca-4bf6-989c-{{Gerrit|a27be2b2d88c}}, use this to unset). - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:58 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 93fa9477-54b8-46e7-8508-{{Gerrit|a2d492528d1f}}, use this to unset). - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f6cf5b29-ce8b-426e-af1c-{{Gerrit|cab74e0c4a3a}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: fe26e612-d76e-4120-b60a-{{Gerrit|4f300add224f}}, use this to unset). - cookbook ran by andrew@buster
* 20:52 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 2abd94be-b6d8-4042-a9ce-{{Gerrit|998e41d932f4}}, use this to unset). - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:51 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 10973c71-30d8-44bf-a310-{{Gerrit|d0c1af76d399}}, use this to unset). - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f8f1b214-ec76-4157-a8e6-{{Gerrit|6374e5d14608}}, use this to unset). - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 95dae613-2b04-472e-a8f9-{{Gerrit|e71c3fbb8d0e}}, use this to unset). - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: c804f4db-ea63-44e9-aff8-{{Gerrit|fea54c238345}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 597a901d-4624-43de-b986-{{Gerrit|c4c6c2fcf80c}}, use this to unset). - cookbook ran by andrew@buster
* 20:03 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:03 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: fdceed0d-6fa0-4806-8fb6-{{Gerrit|3b321a5a1623}}, use this to unset). - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: b0904345-9aa3-4202-b992-{{Gerrit|78644141517c}}, use this to unset). - cookbook ran by andrew@buster
* 19:36 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:31 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:30 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
=== 2022-07-13 ===
* 14:48 bd808: Added Tavvi as member of #acl*wmcs-team
=== 2022-07-12 ===
* 12:04 wm-bot2: Ceph cluster at <nowiki>{</nowiki>self.deployment<nowiki>}</nowiki> set out of maintenance. - cookbook ran by dcaro@vulcanus
* 12:03 wm-bot2: Set the ceph cluster for codfw1dev in maintenance, alert silence ids: db32805f-a033-4e0e-8fc7-{{Gerrit|ed0c2e9f8be1}},f4e698f0-4b51-4b07-9ba8-{{Gerrit|0b296ca1c4fb}},39ad5325-44ed-44d1-bba5-{{Gerrit|91a85c7a8401}},a584a3c6-9dae-41e9-ac82-{{Gerrit|a91cd35af8fb}} - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2005-dev', 'cloudnet2006-dev'] - cookbook ran by dcaro@vulcanus
* 08:37 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:31 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2005-dev,cloudnet2006-dev - cookbook ran by dcaro@vulcanus
=== 2022-07-08 ===
* 15:57 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
=== 2022-07-07 ===
* 07:17 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus
* 07:12 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet ([[phab:T312509|T312509]]) - cookbook ran by dcaro@vulcanus
=== 2022-07-06 ===
* 17:50 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus
=== 2022-07-04 ===
* 13:27 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot2: Rebooted cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:23 wm-bot2: Rebooting all the cloudgw nodes from the eqiad1 deployment: cloudgw1001.eqiad.wmnet,cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:13 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 13:12 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:09 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:08 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:56 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:52 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
* 10:56 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:51 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:47 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 09:07 wm-bot2: Rebooting cloudnet host cloudnet1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:06 wm-bot2: Rebooted cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 09:00 wm-bot2: Rebooting all the cloudnet nodes cloudnet1003,cloudnet1004 - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:59 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:53 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:47 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 08:32 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:25 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 08:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:55 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 07:44 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:36 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:09 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 07:04 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
=== 2022-07-03 ===
* 21:27 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted
=== 2022-07-02 ===
* 11:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
=== 2022-07-01 ===
* 15:35 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:35 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:30 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:29 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 15:13 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:08 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:07 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:42 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:37 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:36 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:32 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 14:19 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:06 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:02 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:58 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:55 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:54 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:52 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:37 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
* 13:34 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
=== 2022-06-30 ===
* 18:17 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 18:13 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:45 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 06:52 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
=== 2022-06-29 ===
* 08:45 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:41 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
=== 2022-06-28 ===
* 13:03 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor [[phab:T301949|T301949]]
=== 2022-06-22 ===
* 16:48 taavi: restart designate-*.service on both cloudservices nodes
* 16:44 taavi: restart nova-conductor on all the cloudcontrol nodes
* 12:50 andrewbogott: rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability
=== 2022-06-21 ===
* 04:48 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
* 04:02 andrewbogott: restarting rabbitmq on cloudcontrol100x (one at a time)
=== 2022-06-17 ===
* 17:53 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. [[phab:T310664|T310664]]
* 06:15 taavi: restart neutron-linuxbridge-agent on cloudvirt1046
=== 2022-06-15 ===
* 11:34 taavi: restart neutron-linuxbridge-agent on cloudvirt1022
=== 2022-06-14 ===
* 16:26 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl
* 14:38 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudceph
* 12:55 wm-bot2: OSDs (['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev']) upgraded successfully B-) ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
* 12:44 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T309786|T309786]]) - cookbook ran by dcaro@vulcanus
=== 2022-06-13 ===
* 11:14 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:08 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:07 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:05 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:04 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 11:03 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
* 09:15 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet ([[phab:T309789|T309789]]) - cookbook ran by dcaro@vulcanus
=== 2022-06-06 ===
* 13:21 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there
=== 2022-06-02 ===
* 07:51 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 [[phab:T309732|T309732]]
* 00:37 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'
=== 2022-06-01 ===
* 17:11 andrewbogott: restarting designate services in cloudservices1xxx hosts
* 16:37 taavi: root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-{{Gerrit|182660616d77}} for [[phab:T309659|T309659]]
=== 2022-05-30 ===
* 17:43 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx
=== 2022-05-29 ===
* 14:55 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage
* 14:15 andrewbogott: restarting rabbitmq on all eqiad1 cloudcontrol nodes (one at a time)
=== 2022-05-25 ===
* 20:01 balloons: clean up cinder backup a bit, restart service due to network outage
* 20:01 balloons: cloudvirt1029 restarted nova due to network outage
=== 2022-05-19 ===
* 15:21 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster
=== 2022-05-18 ===
* 15:42 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build
=== 2022-05-14 ===
* 11:33 taavi: deleted projects 'ores' and 'ores-staging' [[phab:T308102|T308102]]
=== 2022-05-13 ===
* 06:20 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:20 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:16 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:15 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 06:11 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:10 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:09 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:07 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 06:06 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 06:05 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:51 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:50 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:49 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:47 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:45 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:44 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:42 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:41 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:40 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:38 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:37 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:30 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:29 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:19 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:18 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 05:12 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:08 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:02 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 05:01 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:52 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:51 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:48 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:44 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:31 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:30 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:26 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:25 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:24 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:23 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:22 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 04:21 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:18 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:16 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:15 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:37 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:36 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:34 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:27 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:26 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:25 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:55 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:54 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:05 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:04 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 01:23 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:19 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:25 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:21 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:20 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:19 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 00:11 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:07 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
=== 2022-05-12 ===
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:55 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 23:54 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:23 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:20 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 22:17 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 22:13 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:57 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:56 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:55 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:54 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:53 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:52 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:51 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:44 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:42 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:36 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:33 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:31 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:30 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:29 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:28 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:22 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:21 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:18 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 21:11 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:08 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:07 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:05 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:04 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:00 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:59 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:55 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:53 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:52 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:50 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:49 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:48 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:44 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:43 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:31 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:28 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:27 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:23 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:22 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:11 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:10 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:07 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:06 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:58 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:35 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:06 andrewbogott: stopping nfs-server on labstore1004 in preparation for reboot
* 04:12 andrewbogott: rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying
=== 2022-05-11 ===
* 18:48 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:48 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:39 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:38 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:04 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 18:03 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
* 08:56 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 08:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 07:53 dcaro: test
* 04:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 04:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:44 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:43 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:39 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:23 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:22 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:09 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:08 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:04 andrewbogott: reset and recreated the rabbitmq cluster in eqiad1 to get around some broken queues.
* 03:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 03:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 02:25 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 02:25 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
=== 2022-05-10 ===
* 21:43 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:40 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:40 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:35 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 21:32 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 21:32 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:05 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 20:02 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:01 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 20:00 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:57 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:55 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:47 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:46 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:46 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:45 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:44 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:40 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:39 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:39 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:37 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:36 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:36 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:33 wm-bot: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
* 19:33 wm-bot: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:29 wm-bot: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:06 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 19:05 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 19:05 wm-bot: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
* 15:41 andrewbogott: rebooting cloud*-dev for [[phab:T307668|T307668]]
* 13:59 taavi: manually attached [[User:Dreamy Jazz]] to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)
=== 2022-05-07 ===
* 01:33 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:32 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:30 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
* 01:21 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 01:18 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
=== 2022-05-03 ===
* 20:38 andrewbogott: upgrading clouddb2001-dev in place
* 18:18 taavi: updated 'puppet-enc' endpoints on the keystone catalog to use https and port 443
=== 2022-05-02 ===
* 16:56 dcaro: rebooting cloudmetrics1001
=== 2022-04-29 ===
* 14:22 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions [[phab:T277653|T277653]] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline
=== 2022-04-27 ===
* 14:51 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud
* 14:50 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:46 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:41 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:35 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:31 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:27 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:13 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:09 wm-bot: Finished rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:05 wm-bot: Finished rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:01 wm-bot: Finished rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:57 wm-bot: Finished rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:44 wm-bot: Rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:43 wm-bot: Finished rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:39 wm-bot: Finished rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:35 wm-bot: Finished rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:31 wm-bot: Finished rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:26 wm-bot: Finished rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:14 wm-bot: Finished rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:10 wm-bot: Finished rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:05 wm-bot: Finished rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Finished rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:57 wm-bot: Finished rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Finished rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:50 wm-bot: Finished rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:46 wm-bot: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024 - cookbo
* 12:15 wm-bot: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] - cookbook ran by dcaro@vulcanus
* 12:15 wm-bot: Finished rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:12 wm-bot: Finished rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:09 wm-bot: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 12:07 wm-bot: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
* 12:05 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 12:02 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:59 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:56 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by dcaro@vulcanus
* 11:55 wm-bot: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:52 wm-bot: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:47 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 11:43 wm-bot: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by dcaro@vulcanus
=== 2022-04-26 ===
* 10:36 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup
=== 2022-04-22 ===
* 10:33 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity
=== 2022-04-21 ===
* 05:38 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]
=== 2022-04-19 ===
* 15:29 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host
=== 2022-04-18 ===
* 15:23 andrewbogott: reimaging cloudvirt1020, leaving VMs in place
* 13:40 andrewbogott: shutting down many codfdfw1dev servers (including network infra!) for [[phab:T305469|T305469]]
=== 2022-04-14 ===
* 20:14 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections
=== 2022-04-13 ===
* 22:01 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections
=== 2022-04-11 ===
* 15:59 taavi: created cloudinfra.wmcloud.org zone
=== 2022-04-09 ===
* 19:55 andrewbogott: reimaging cloudbackup1001-dev to bullseye
* 19:37 taavi: add 'puppet-enc' service & endpoint to keystone [[phab:T274666|T274666]]
* 19:25 andrewbogott: reimaging cloudbackup1002-dev to bullseye
=== 2022-04-07 ===
* 12:51 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T305631|T305631]]) - cookbook ran by arturo@nostromo
=== 2022-04-06 ===
* 09:12 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) ([[phab:T305157|T305157]])
* 08:45 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:42 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])
* 08:24 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudvirt2003-dev ([[phab:T305157|T305157]])
* 08:20 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudcontrol servers ([[phab:T305157|T305157]])
=== 2022-03-30 ===
* 11:20 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - [[phab:T285461|T285461]]
=== 2022-03-29 ===
* 10:02 dcaro: restarting keystone ([[phab:T304918|T304918]])
=== 2022-03-23 ===
* 22:53 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:38 wm-bot: Drained 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:12 wm-bot: Draining 'cloudvirt1045.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:08 wm-bot: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:07 wm-bot: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1044.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:06 wm-bot: Draining 'cloudvirt1043.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:54 wm-bot: Drained 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:19 wm-bot: Draining 'cloudvirt1042.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Drained 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:12 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:09 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:07 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 21:04 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:55 wm-bot: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:54 wm-bot: Draining 'cloudvirt1041.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:30 wm-bot: Drained 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:14 wm-bot: Draining 'cloudvirt1039.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:44 wm-bot: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:43 wm-bot: Draining 'cloudvirt1038.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:19 wm-bot: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Draining 'cloudvirt1037.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:13 wm-bot: Drained 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:02 wm-bot2: Testing wm-bot relay to #wikimedia-cloud-feed
* 17:55 wm-bot: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:54 wm-bot: Draining 'cloudvirt1036.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:04 wm-bot: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Draining 'cloudvirt1035.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:03 wm-bot: Drained 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:51 wm-bot: Drained 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:37 wm-bot: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1034.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Draining 'cloudvirt1033.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:01 wm-bot: Drained 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 15:00 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:57 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:44 wm-bot: Drained 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:35 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:34 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:32 wm-bot: Drained 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:20 wm-bot: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:19 wm-bot: Draining 'cloudvirt1031.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:18 wm-bot: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 14:17 wm-bot: Draining 'cloudvirt1030.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:54 taavi: restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change
* 13:43 wm-bot: Drained 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:23 wm-bot: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:22 wm-bot: Draining 'cloudvirt1029.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
=== 2022-03-22 ===
* 22:59 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 22:58 wm-bot: Draining 'cloudvirt1027.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
=== 2022-03-17 ===
* 01:09 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:53 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 00:52 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
=== 2022-03-15 ===
* 20:58 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:36 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:14 wm-bot: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 13:14 wm-bot: Unsetting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:32 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
* 10:30 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
=== 2022-03-14 ===
* 21:24 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:59 wm-bot: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Setting cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:58 wm-bot: Draining 'cloudvirt1025.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 20:02 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:59 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:16 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 19:13 wm-bot: Drained 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:56 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Setting cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:55 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:53 wm-bot: Drained 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:52 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:51 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:50 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 11:48 dcaro: rebased cookbooks on latest master, make sure you pull before sending new patches
=== 2022-03-08 ===
* 18:29 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:29 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:23 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:21 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:18 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 18:17 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:27 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:18 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 17:15 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:48 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:47 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:36 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:08 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 16:07 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. ([[phab:T281276|T281276]]) - cookbook ran by andrew@buster
* 13:11 arturo: [codfw1dev] rebooting cloudservices servers for [[phab:T303179|T303179]]
* 13:07 arturo: [codfw1dev] rebooting cloudvirt servers for [[phab:T303179|T303179]]
* 13:06 arturo: [codfw1dev] rebooting cloudnet servers for [[phab:T303179|T303179]]
* 12:55 arturo: [codfw1dev] rebooting cloudcontrol servers for [[phab:T303179|T303179]]
=== 2022-03-03 ===
* 08:49 taavi: deploying cloudmetrics grafana to grafana 8, [[phab:T282863|T282863]]
=== 2022-03-02 ===
* 09:06 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347
=== 2022-02-28 ===
* 15:30 dcaro: cleaning up leftover snapshots from failed backups of the maps volume ([[phab:T302720|T302720]])
=== 2022-02-24 ===
* 17:04 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'
* 15:42 dcaro: stopping and starting mariadb on cloudcontrol1003 ([[phab:T302146|T302146]])
* 10:37 arturo: [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install ([[phab:T302482|T302482]])
=== 2022-02-23 ===
* 20:39 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account [[phab:T295246|T295246]]
* 17:40 andrewbogott: restarting lots of openstack services to try to clear up the mess that is [[phab:T236101|T236101]]
* 12:13 arturo: cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done ([[phab:T302382|T302382]])
* 10:14 arturo: cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet
* 10:05 dcaro: Deleting stuck novafullstack servers, to let the service create new ones ([[phab:T302369|T302369]])
* 09:56 arturo: neutron agent-delete bad663b3-fd25-4393-a546-{{Gerrit|4b1b4bdec4db}} (Linux bridge agent {{!}} cloudvirtan1001)
* 09:56 arturo: neutron agent-delete 1071c198-ed57-4b5a-9439-{{Gerrit|30e66a31aa69}} (Linux bridge agent {{!}} cloudvirtan1005)
* 09:55 arturo: neutron agent-delete 2eeef198-8af7-4e5d-bd73-{{Gerrit|e14a2a8d2404}} (Linux bridge agent {{!}} cloudvirtan1004)
* 09:55 arturo: neutron agent-delete afe173eb-35ba-444a-9960-{{Gerrit|899629786d2f}} (Linux bridge agent {{!}} cloudvirtan1003)
* 09:54 arturo: neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-{{Gerrit|92bfbe8d1a56}} (Linux bridge agent {{!}} cloudvirtan1002)
* 09:39 dcaro: restarting neutron-api cloudcontrol1003 to see if the agent status update starts working ([[phab:T302369|T302369]])
* 09:38 dcaro: restarting neutron-dhcp-agent on cloudnet1003 ([[phab:T302369|T302369]])
=== 2022-02-22 ===
* 22:10 andrewbogott: raising project 'maps' quota by two tb -- [[phab:T300160|T300160]]
* 09:24 arturo: restarting mariadb @ cloudcontrol1003 ([[phab:T302146|T302146]])
* 09:13 arturo: restarting mariadb @ cloudcontrol1004 ([[phab:T302146|T302146]])
=== 2022-02-18 ===
* 21:57 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend.  Everything there should be stable and fine save rabbit which needs an upgrade.
* 21:30 andrewbogott: rebooting cloudcontrol1003 because rabbit is freaking out
* 17:25 andrewbogott: in-place upgrade of cloudcontrol1004 to bullseye -- [[phab:T281276|T281276]]
* 12:34 arturo: manually install prometheus-openstack-exporter on cloudcontrol1005 ([[phab:T302050|T302050]])
=== 2022-02-17 ===
* 23:02 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 [[phab:T281276|T281276]]
=== 2022-02-15 ===
* 14:15 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account [[phab:T295246|T295246]]
=== 2022-02-04 ===
* 10:12 arturo: restart backup_vms service in cloudvirt1024 ([[phab:T300956|T300956]])
=== 2022-02-03 ===
* 08:21 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing "performing a change every puppet run" alert)
=== 2022-02-02 ===
* 02:36 andrewbogott: restarting mariadb on cloudcontrol1004
=== 2022-01-31 ===
* 10:15 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)
=== 2022-01-29 ===
* 18:24 taavi: delete 2 puppet prefixes in a weird state [[phab:T299750|T299750]]
=== 2022-01-27 ===
* 13:24 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service ([[phab:T300138|T300138]])
=== 2022-01-26 ===
* 19:09 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004
* 18:57 andrewbogott: restarting mariadb on cloudcontrol1004
=== 2022-01-25 ===
* 10:49 arturo: made cloudmetrics1001/1002 primary/backup respectively ([[phab:T299744|T299744]], [[phab:T297814|T297814]], [[phab:T300011|T300011]])
=== 2022-01-19 ===
* 16:38 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud
=== 2022-01-05 ===
* 03:11 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498
=== 2022-01-04 ===
* 12:44 dcaro: increasing the size_limit for labs ldap servers
=== 2021-12-26 ===
* 16:55 majavah: run attachLdapUser.php on wikitech for developer account "Karthiksripal"
=== 2021-12-24 ===
* 22:51 majavah: ran the wikireplica dns script on s5 [[phab:T298303|T298303]]
=== 2021-12-23 ===
* 21:42 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies
=== 2021-12-21 ===
* 10:39 arturo: dropped egress NAT exceptions for WMF apt repos, [[phab:T298042|T298042]]
=== 2021-12-15 ===
* 12:44 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed ([[phab:T297454|T297454]])
=== 2021-12-14 ===
* 10:26 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/<canary_image_id>) to the root disk on cloudvirt-wdqs1001 to temporary free some space ([[phab:T297454|T297454]])
=== 2021-12-13 ===
* 18:08 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:50 wm-bot: Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Setting cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:49 wm-bot: Draining 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:44 wm-bot: Drained 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:30 wm-bot: Set cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Setting cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 17:30 wm-bot: Draining 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
* 17:13 wm-bot: Drained 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:50 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:47 wm-bot: Draining 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
* 16:44 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
* 16:43 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
=== 2021-12-03 ===
* 18:56 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
* 10:49 majavah: deleting dbbackups-dashboard project [[phab:T296992|T296992]]
=== 2021-12-02 ===
* 01:17 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:56 wm-bot: Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:56 wm-bot: Draining 'cloudvirt1028.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:50 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:50 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:28 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 00:03 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 00:02 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
=== 2021-12-01 ===
* 23:59 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
* 23:59 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. ([[phab:T296790|T296790]]) - cookbook ran by andrew@buster
* 23:54 andrewbogott: *correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]
* 23:53 andrewbogott: adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around [[phab:T296790|T296790]] and [[phab:T296792|T296792]]
=== 2021-11-28 ===
* 17:48 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for [[phab:T296592|T296592]].  It will need to be moved back after the raid is rebuilt.
=== 2021-11-21 ===
* 07:19 dcaro_away: restarting designate-sink with some extra logs in it ([[phab:T296144|T296144]])
=== 2021-11-17 ===
* 15:48 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
* 15:39 andrewbogott: sudo cumin  "cloud*" 'apt-get update -y --allow-releaseinfo-change'
* 15:26 andrewbogott: updated mariadb packages on codfw1dev cloudcontrols to 1:10.3.31-0+deb10u1
=== 2021-11-12 ===
* 13:31 arturo: restarting glance-api services to make sure they work with new ceph auth creds ([[phab:T293752|T293752]])
=== 2021-11-08 ===
* 21:50 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 [[phab:T216481|T216481]]
* 20:07 andrewbogott: depooling clouddb1013 for maintain_views attempt
* 10:54 arturo: [codfw1dev] create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]
* 10:34 arturo: create service account `srv-networktests` following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for [[phab:T294955|T294955]]
=== 2021-11-05 ===
* 11:18 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:17 wm-bot: Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:15 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:12 wm-bot: Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
=== 2021-11-04 ===
* 16:39 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:39 wm-bot: Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:37 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:34 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:33 wm-bot: Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:17 wm-bot: Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:16 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:13 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:12 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 16:00 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:26 wm-bot: Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:23 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:20 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:19 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
* 11:16 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster ([[phab:T295012|T295012]]) - cookbook ran by arturo@endurance
=== 2021-11-03 ===
* 17:22 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev ([[phab:T294956|T294956]])
* 11:45 arturo: [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev ([[phab:T294853|T294853]], [[phab:T291813|T291813]])
=== 2021-11-02 ===
* 10:54 arturo: rebooting cloudnet1004/1003 for [[phab:T291813|T291813]]
* 10:43 arturo: [codfw1dev] rebooting cloudgw200[12]-dev for [[phab:T291813|T291813]]
=== 2021-10-24 ===
* 00:47 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738
=== 2021-10-21 ===
* 10:19 arturo: drop firewall exception on core routers for wiki replicas legacy setup ([[phab:T293897|T293897]])
* 10:12 arturo: drop NAT exception for wiki replicas legacy setup ([[phab:T293897|T293897]])
=== 2021-10-20 ===
* 21:06 andrewbogott: creating cloudinfra-nfs project [[phab:T293936|T293936]]
=== 2021-10-18 ===
* 19:21 andrewbogott: also ticked the 'admin' box on wikitech for majavah [[phab:T292827|T292827]]
* 18:58 andrewbogott: granting majavah 'admin' role in the 'admin' project and also in the default domain.  [[phab:T292827|T292827]]
=== 2021-10-14 ===
* 12:28 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB ([[phab:T292546|T292546]])
=== 2021-10-13 ===
* 10:46 arturo: updating python3-neutron across the fleet ([[phab:T292936|T292936]])
=== 2021-10-12 ===
* 09:06 dcaro: upgrading eqiad cloudnet hosts neutron packages ([[phab:T292936|T292936]])
* 08:57 dcaro: upgrading codfw cloudnet hosts neutron packages ([[phab:T292936|T292936]])
=== 2021-10-05 ===
* 09:39 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) [[phab:T291257|T291257]]
=== 2021-09-30 ===
* 14:50 andrewbogott: sudo cumin  "cloud*" "ps -ef {{!}} grep nslcd && service nslcd restart" and sudo cumin  "lab*" "ps -ef {{!}} grep nslcd && service nslcd restart"  [[phab:T292202|T292202]]
* 14:43 andrewbogott: ran sudo cumin --force --timeout 500 -o json  "A:all" "ps -ef {{!}} grep nslcd && service nslcd restart" to get nslcd happy again [[phab:T292202|T292202]]
=== 2021-09-29 ===
* 09:41 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured ([[phab:T291257|T291257]])
=== 2021-09-28 ===
* 16:23 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly [[phab:T291963|T291963]]
* 16:21 bstorm: powering on clouddb1020 via remote console [[phab:T291963|T291963]]
* 15:58 bstorm: depooled clouddb1020 for repair [[phab:T291961|T291961]]
* 12:40 dcaro: Merged change on sssd for bullseye cloud hosts ([[phab:T291585|T291585]])
* 11:30 arturo: [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org ([[phab:T291257|T291257]])
=== 2021-09-27 ===
* 10:07 arturo: cloudcontrol1004 apparently healthy [[phab:T291446|T291446]]
* 09:25 arturo: rebooting cloudcontrol1004 for [[phab:T291446|T291446]]
=== 2021-09-24 ===
* 13:02 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
* 13:00 arturo: [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict
=== 2021-09-21 ===
* 12:13 arturo: [codfw1dev] trying to create a manila service image ([[phab:T291257|T291257]])
* 11:45 arturo: [codfw1dev] created rabbitmq user ([[phab:T291257|T291257]])
* 11:32 arturo: [codfw1dev] populated manila DB & created service endpoints ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] give manila user admin role @ manila project ([[phab:T291257|T291257]])
* 11:06 arturo: [codfw1dev] created manila project ([[phab:T291257|T291257]])
* 10:57 arturo: [codfw1dev] created manila user @ labtestwikitech ([[phab:T291257|T291257]])
* 10:49 arturo: [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) [[phab:T291257|T291257]]
=== 2021-09-20 ===
* 23:08 bstorm: ran `echo check > /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
* 22:48 andrewbogott: stopped puppet & mariadb on cloudcontrol1004; it was flapping
* 22:44 andrewbogott: sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there
* 21:57 andrewbogott: moving cloudvirt1043 into the 'nfs' aggregate for [[phab:T291405|T291405]]
=== 2021-09-17 ===
* 11:35 arturo: [codfw1dev] install manila on cloudcontrol2001-dev ([[phab:T291257|T291257]])
=== 2021-09-16 ===
* 15:56 bstorm: removing downtime for labstore1005 so we'll know if it has another issue [[phab:T290318|T290318]]
=== 2021-09-09 ===
* 22:03 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working [[phab:T290630|T290630]]
* 03:15 bstorm: resetting swap on clouddb1017 [[phab:T290630|T290630]]
* 03:08 andrewbogott: stopping maintain-dbusers on labstore1004 for help diagnosing [[phab:T290630|T290630]]
=== 2021-09-03 ===
* 15:34 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 [[phab:T290318|T290318]]
* 15:24 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 [[phab:T290318|T290318]]
* 15:20 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 [[phab:T290318|T290318]]
=== 2021-08-30 ===
* 16:16 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
* 16:16 wm-bot: Added OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:10 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:07 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
=== 2021-08-27 ===
* 18:57 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye
=== 2021-08-25 ===
* 14:45 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 14:42 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:41 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
=== 2021-08-19 ===
* 17:39 bstorm: restarting glance image backup to try and clear the page
=== 2021-08-18 ===
* 16:21 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
* 16:21 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:17 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:16 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:15 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
* 14:47 andrewbogott: adding clouvirt1038 to the ceph aggregate, removing from the maintenance aggregate [[phab:T276922|T276922]]
=== 2021-08-17 ===
* 15:11 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- [[phab:T287838|T287838]]
=== 2021-08-11 ===
* 13:51 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:48 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 13:47 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
=== 2021-08-10 ===
* 15:15 andrewbogott: restarting all designate services in eqiad1
* 15:04 andrewbogott: restarting designate-sink in eqiad1; it's complaining about rabbit but I don't want to restart rabbit yet
=== 2021-08-05 ===
* 09:37 dcaro: Taking one osd daemon down ot codfw cluster ([[phab:T288203|T288203]])
=== 2021-08-04 ===
* 19:20 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki
=== 2021-08-03 ===
* 17:40 bstorm: rerunning the glance backup script after failure
=== 2021-07-31 ===
* 00:10 andrewbogott: "systemctl reset-failed cloud-init.service" on all VMs for [[phab:T287309|T287309]]
* 00:08 andrewbogott: "systemctl reset-failed cloud-final.service" on all VMs for [[phab:T287309|T287309]]
=== 2021-07-27 ===
* 21:32 andrewbogott: putting cloudvirt1012 back into service [[phab:T286748|T286748]]
* 20:52 andrewbogott: draining VMs off of cloudvirt1012 so we can replace the battery for [[phab:T286748|T286748]]
* 15:15 andrewbogott: "rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list" cloud-wide
=== 2021-07-23 ===
* 15:22 bstorm: update wikireplicas-dns for s7 fix for web replicas
=== 2021-07-20 ===
* 17:07 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 15:45 arturo: failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417
* 00:10 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater
=== 2021-07-19 ===
* 22:05 bstorm: set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 [[phab:T286599|T286599]]
* 20:40 andrewbogott: reloading haproxy on dbproxy1018 for [[phab:T286598|T286598]]
* 13:50 andrewbogott: upgrading mariadb to 10.3.29 on all cloudcontrols
=== 2021-07-16 ===
* 09:55 dcaro: checking HP raid issues on coludvirt1012 ([[phab:T286766|T286766]])
=== 2021-07-14 ===
* 21:08 andrewbogott: restarting lots of openstack services while trying to resolve [[phab:T286675|T286675]]
* 12:17 dcaro: doing ceph outage tests on codfw1 (fyi)
=== 2021-07-13 ===
* 10:57 dcaro: enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128
=== 2021-07-02 ===
* 10:12 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:12 wm-bot: Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:10 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:07 wm-bot: Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:05 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 10:02 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:01 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 09:13 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
=== 2021-07-01 ===
* 16:27 bstorm: failed over cloudstore1009 to cloudstore1008 [[phab:T224747|T224747]]
* 16:18 bstorm: downtimed cloudstore1008 and cloudstore1009 to fail over [[phab:T224747|T224747]]
* 14:25 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:25 wm-bot: Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:24 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:21 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:20 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 14:17 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:16 wm-bot: Added new OSD node cloudcephosd1016.eqiad.wmnet ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 11:13 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:58 dcaro: rebooting cloudcephosd1016 ([[phab:T285858|T285858]])
* 10:47 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:44 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:41 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
* 10:40 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster ([[phab:T285858|T285858]]) - cookbook ran by dcaro@vulcanus
=== 2021-06-30 ===
* 21:48 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration
=== 2021-06-25 ===
* 15:28 andrewbogott: restarting openstack services on cloudcontrol1005
* 09:16 arturo: icinga downtime cloudcontrols for 2h
* 08:20 dcaro: restarting rabbitmq on cloudcontrol100<nowiki>{</nowiki>3,4<nowiki>}</nowiki>
=== 2021-06-21 ===
* 13:54 dcaro: puppet fix merged and deployed, servers are back to normal
* 13:20 dcaro: merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer)
=== 2021-06-20 ===
* 22:21 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster
=== 2021-06-15 ===
* 01:18 bstorm: running a modified version of the prometheus dir size cron in screen [[phab:T284964|T284964]]
=== 2021-06-14 ===
* 10:13 dcaro: setting ssd to debug mode on tools-sgeexec-0917 ([[phab:T284130|T284130]])
=== 2021-06-10 ===
* 10:58 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:58 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:55 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:52 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:49 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:48 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:45 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:42 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 10:39 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:39 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:38 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:35 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:32 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:29 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:26 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 09:24 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
=== 2021-06-09 ===
* 17:33 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy ([[phab:T281881|T281881]])
* 13:30 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:30 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:27 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:24 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:21 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 13:01 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 12:53 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
=== 2021-06-08 ===
* 23:19 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 ([[phab:T281881|T281881]])
* 21:08 bstorm: downtiming grafana-labs for maintenance
* 16:28 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:27 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:24 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:22 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:21 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:18 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 16:17 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] ([[phab:T281248|T281248]]) - cookbook ran by dcaro@vulcanus
* 15:03 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:23 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
* 14:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
=== 2021-06-07 ===
* 14:27 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate  [[phab:T281399|T281399]]
=== 2021-06-01 ===
* 13:12 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
* 09:57 arturo: fix PTR record for 185.15.56.1 ([[phab:T284025|T284025]])
* 09:56 arturo: fix PTR record for 185.15.56.1 ([[phab:T248025|T248025]])
=== 2021-05-27 ===
=== 2021-05-27 ===
* 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus
* 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus

Revision as of 10:48, 27 September 2022

2022-09-27

  • 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
  • 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
  • 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
  • 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
  • 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@wmf3169
  • 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
  • 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
  • 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
  • 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
  • 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
  • 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
  • 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
  • 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

2022-09-26

  • 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:31 andrewbogott: rebooting cloudvirt1028 for T317391
  • 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-eb0413dfb9d0, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-d2cdd8abb2bf, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-06b6f53fd899, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-36bc7f79b4d5, use this to unset). - cookbook ran by andrew@buster
  • 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-35e675abce32, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-66f54ba9801c, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-58aaa9c8e021, use this to unset). - cookbook ran by andrew@buster
  • 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-23c636d6554e, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-1aa1cfe4c5af, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-077ffa1c1c5e, use this to unset). - cookbook ran by andrew@buster
  • 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-ef7785a4af4a, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-1e768b602a1b, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-074d82582cb4, use this to unset). - cookbook ran by andrew@buster
  • 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-cd1cf910f8e1, use this to unset). - cookbook ran by andrew@buster
  • 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-29b7699b692f, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-5c41992192b4, use this to unset). - cookbook ran by andrew@buster
  • 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-6ad4dcb2a105, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-8a6d481dea7d, use this to unset). - cookbook ran by andrew@buster
  • 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-8f43c253e499, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
  • 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
  • 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
  • 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin (T318047)
  • 10:14 arturo: deployed new version of maintain-dbusers (T318047)

2022-09-25

  • 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-3993c1e8b593, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-8663db77ce66, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-e5625d209b57, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-b3a64bd6b18d, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-afd57ae1ca55, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

2022-09-24

  • 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
  • 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022

2022-09-22

  • 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-538741a2f801, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-c41d1e1c8c8b, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-6efde61db3a3, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-ff23055a0527, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-fd671c398642, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-bc66789d37c8, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-cce5edaa1f08, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-eeee37007d14, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-76d499eccfa1, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-677853fb9ef7, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-66df673c7ed8, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-2ed2e474c89f, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-265f6386f396, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-90ab1f492aa8, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-c571b3df2945, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-513767f95b1d, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-b0005c8164d1, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-0d513ad7acb5, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-2f49d1033f78, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-341981b3887f, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-139b4d119ef8, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-47d87262d0ed, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

2022-09-20

  • 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-4f52f61d503f, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-7e9d518be217, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-2b18032bbf3c, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-ee6bbbf029fa, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-a356c39f3330, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-01795b1e62ce, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-5a7dc56060b6, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-303d16801e28, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-e20cbb877755, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-514ca5cdc76d, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:05 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:01 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:01 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 8f2d2d32-e99e-4e2c-9472-ca33b577d01f, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:00 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:00 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:00 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:00 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 66ee7d87-cee1-4c8f-b5c3-5f58a6cea336, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:59 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:59 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:58 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:58 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:58 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 69035b38-9910-46f1-9582-3105c1ba4d16, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:57 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:57 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 19:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 19:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:52 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 5a94c9d1-c956-4b03-a92b-b112810906a4, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:51 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:51 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:50 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 216a9120-4b9a-4be3-99db-3fd5fdc25146, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:49 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:49 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:48 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 46fb1964-3cf4-47fd-ad79-475f33c8ed38, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:48 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:48 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b04a4f8c-db82-4258-8608-2a78ec9193ed, use this to unset). - cookbook ran by andrew@buster
  • 19:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:46 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:46 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 7ae7c258-ecb8-47d1-a991-3fa617e305bf, use this to unset). - cookbook ran by andrew@buster
  • 19:45 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:44 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 377fb0e5-132e-4e7a-9079-37937c3d42bf, use this to unset). - cookbook ran by andrew@buster
  • 19:44 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 6c4222ae-796e-4890-8820-9e7ce5438f2a, use this to unset). - cookbook ran by andrew@buster
  • 19:43 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:43 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:38 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:34 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 19:34 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 19:31 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: eff98e2e-ae99-41d0-a68e-671d941fcf21, use this to unset). - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5310f1eb-0425-4f59-8fb5-1d9c6f6d2f23, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:29 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:25 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: d8ab4682-026a-4ddc-bfc5-48166bd9789a, use this to unset). - cookbook ran by andrew@buster
  • 19:24 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:23 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 19:23 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 19:19 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:19 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: eb2f9d94-8ea5-48d5-a336-97dd407b1ce3, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 19:19 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:19 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:18 wm-bot2: Drained 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:14 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: 305fe2d7-913e-4638-95c2-4e810dc6b2a3, use this to unset). - cookbook ran by andrew@buster
  • 19:14 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 5bb3a66b-9450-4230-8d09-d78473d72ba2, use this to unset). - cookbook ran by andrew@buster
  • 19:13 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:13 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:07 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: b1e39b49-3a88-4e42-913c-9892eafad447, use this to unset). - cookbook ran by andrew@buster
  • 19:06 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:05 andrewbogott: putting cloudvirt1049-1052 into 'ceph' pool, taking out of 'spare' pool. cloudvirt1053 will remain our only spare.
  • 19:02 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 19:02 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 19:00 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 19:00 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:59 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:59 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:58 wm-bot2: Drained 'cloudvirt1049.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:58 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 8f24ea75-e107-4a13-8bd6-950b941b8e03, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:58 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:58 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:57 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:57 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:56 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:56 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 5c3e8fa4-cd1e-43f4-a423-ba7fe2459d4e, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:55 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:55 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: e3fe22d6-932c-4502-8838-82cdc4c8f1c6, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:55 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:55 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:54 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:54 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:53 wm-bot2: Safe reboot of 'cloudvirt1053.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 18:53 wm-bot2: Unset cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 18:52 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:52 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: fd824b92-b6ff-4631-a2d4-debb25d48223, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:52 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:52 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:51 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:49 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 52d5cd5b-0f56-453a-85f8-0b5e656ae7f9, use this to unset). - cookbook ran by andrew@buster
  • 18:49 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:49 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: b92f66a3-a487-4515-b68b-1c257a57b893, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 18:48 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:48 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 18:48 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

2022-09-19

  • 20:07 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
  • 20:07 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 4bfca6cd-dfbc-4a79-9203-b432bfeee5c2, use this to unset). (T317391) - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
  • 19:41 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:21 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 70479325-609b-4349-9094-739eb9b3bb30, use this to unset). - cookbook ran by andrew@buster
  • 19:21 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster

2022-09-14

  • 16:57 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 16:53 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 16:53 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 16:53 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 11:01 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 10:58 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 10:57 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
  • 10:57 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
  • 10:09 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
  • 10:09 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
  • 10:06 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
  • 10:06 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
  • 09:46 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:43 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:43 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:43 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-09-13

  • 12:16 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
  • 12:16 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
  • 12:11 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
  • 12:11 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
  • 12:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
  • 12:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
  • 10:40 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:37 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:37 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:36 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-09-10

  • 15:37 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

2022-09-08

  • 19:12 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

2022-09-07

  • 10:18 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:14 wm-bot2: Finished rebooting node cloudcephosd1032.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:10 wm-bot2: Rebooting node cloudcephosd1032.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:10 wm-bot2: Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:10 wm-bot2: Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:41 dhinus: Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)

2022-08-30

  • 14:59 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for T296561
  • 10:43 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:39 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:38 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:38 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:39 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:14 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:11 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:10 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:10 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-08-25

  • 15:14 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 15:14 wm-bot2: Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 15:02 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 14:59 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 14:58 wm-bot2: Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 14:58 wm-bot2: Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-08-24

  • 22:07 andrewbogott: replaced cloudservices1003 with cloudservices1005 T304888
  • 10:45 wm-bot2: Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:45 wm-bot2: Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:37 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:34 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:33 wm-bot2: Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 10:33 wm-bot2: Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-08-23

  • 13:46 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:46 wm-bot2: Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:27 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:24 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:22 wm-bot2: Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:22 wm-bot2: Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-08-21

  • 21:12 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

2022-08-20

  • 07:39 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover (T315718)
  • 07:23 dcaro_away: cloudvirt1023 seems to have gotten some hardware issue from racadm lclog view "System CPU Resetting.", rebooting and doing memory checks (T315718)

2022-08-19

  • 17:06 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes T310795

2022-08-18

  • 13:25 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:25 wm-bot2: Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:15 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:12 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:10 wm-bot2: Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 13:10 wm-bot2: Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 07:29 dcaro: Starting up all the osd daemons on cloudcephosd1025 (T314870)

2022-08-17

  • 10:50 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@foz
  • 10:49 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@foz
  • 10:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@foz
  • 10:37 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@foz
  • 10:37 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@foz
  • 10:37 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@foz
  • 09:50 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@foz
  • 09:49 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@foz
  • 09:49 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@foz
  • 09:16 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:16 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro
  • 09:16 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

2022-08-16

  • 22:39 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate
  • 17:41 andrewbogott: removing cloudvirt1025 from the 'ceph' aggregate and adding it to the 'maintenance' aggregate
  • 17:40 andrewbogott: reimaging cloudvirt1025 after I accidentally deleted the hw raid
  • 17:38 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1003@rbd#RBD --newhost cloudcontrol1005@rbd#RBD
  • 17:37 andrewbogott: root@cloudcontrol1005:~# cinder-manage volume update_host --currenthost cloudcontrol1004@rbd#RBD --newhost cloudcontrol1006@rbd#RBD
  • 16:26 wm-bot2: Ceph cluster at eqiad1 set out of maintenance. - cookbook ran by dcaro@vulcanus
  • 15:43 wm-bot2: Restarting the osd daemons from nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,c
  • 15:42 wm-bot2: Finished restarting all the OSD daemons from the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
  • 15:38 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
  • 13:08 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
  • 13:07 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
  • 13:02 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
  • 12:59 wm-bot2: Restarting the osd daemons from nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus

2022-08-14

  • 18:36 taavi: deleted the http keystone endpoints from the keystone service catalog

2022-08-11

  • 13:57 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.
  • 08:42 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 08:42 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 08:42 wm-bot2: Added OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 08:40 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 08:36 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 08:36 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 08:36 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@MacBook-Pro.station

2022-08-10

  • 13:10 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 13:06 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 13:06 wm-bot2: Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@MacBook-Pro.station
  • 13:06 wm-bot2: Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@MacBook-Pro.station

2022-08-04

  • 17:16 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling
  • 16:32 taavi: restart neutron-l3-agent to pick up rabbit config changes
  • 15:12 andrewbogott: stopping rabbitmq on cloudcontrol1xxx
  • 09:57 taavi: stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom

2022-08-03

  • 20:55 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service
  • 20:41 andrewbogott: restarting neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq

2022-08-02

2022-07-27

2022-07-25

  • 13:43 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

2022-07-22

  • 16:41 taavi: depool cloudweb1003/1004 since horizon seems to be having issues
  • 16:22 taavi: pooling cloudweb1003/1004 now that grant issues are sorted

2022-07-21

  • 18:26 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes
  • 18:06 andrewbogott: pooling cloudweb1003 and 1004 for wikitech, horizon, striker

2022-07-20

  • 18:02 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400)
  • 17:45 bd808: `sudo service striker restart` on labweb1002
  • 17:43 bd808: `sudo service striker restart` on labweb1001
  • 17:10 dcaro: things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 (T313400)
  • 16:26 dcaro: things seem stable, trying to bring up a third, cloudcontrol1005 (T313400)
  • 15:51 dcaro: things seem stable now with one rabbit node, trying to bring up a second (T313400)
  • 14:16 dcaro: stopping rabbin on cloudcontrol1004, leaving only 1003 alive (T313400)
  • 13:17 dcaro: restarting the whole rabbit cluster (T313400)

2022-07-19

  • 16:30 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 16:30 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 16:26 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 16:18 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 16:18 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 16:14 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 16:01 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 16:01 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:57 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:57 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-2b010960ed6a, use this to unset). - cookbook ran by andrew@buster
  • 15:56 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:56 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:56 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 15:56 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:52 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:49 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-363a7dea8d69, use this to unset). - cookbook ran by andrew@buster
  • 15:47 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-911b14f2314d, use this to unset). - cookbook ran by andrew@buster
  • 15:47 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-0d4638acc9c3, use this to unset). - cookbook ran by andrew@buster
  • 15:46 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:46 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:46 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:46 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:46 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:46 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:45 andrewbogott: adding new hosts to the 'ceph' aggregate: cloudvirt1046, 1047, 1048
  • 15:44 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 15:44 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:42 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 15:42 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:40 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:38 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:37 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 15:37 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:33 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 32d9fdb7-6c42-4cf4-9950-6e2442255161, use this to unset). - cookbook ran by andrew@buster
  • 15:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:16 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 56ce1388-792c-442c-bb89-e4c3869b0acf, use this to unset). - cookbook ran by andrew@buster
  • 15:15 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:15 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:14 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3f84bf2e-24de-4ebe-8117-0f4a5de29d09, use this to unset). - cookbook ran by andrew@buster
  • 15:14 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:14 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:13 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 15:13 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:12 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 5418e923-997c-49ae-b937-35fc0c3038eb, use this to unset). - cookbook ran by andrew@buster
  • 15:12 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: c3dfdc2b-0725-4d01-87f2-440a5ac4694c, use this to unset). - cookbook ran by andrew@buster
  • 15:11 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:11 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:11 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:11 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:09 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:04 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 15:04 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 15:00 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:54 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 14:54 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 14:50 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:46 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 49dbf7de-c58a-4b68-bc7b-f001a047f4c7, use this to unset). - cookbook ran by andrew@buster
  • 14:46 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:46 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:44 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: a4e95168-39b5-452d-8fce-7875ae44a62b, use this to unset). - cookbook ran by andrew@buster
  • 14:44 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:44 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:43 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: b46234ee-9900-4708-a815-4324379be48c, use this to unset). - cookbook ran by andrew@buster
  • 14:43 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 14:43 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:42 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:39 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 14:39 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 14:38 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:38 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 12bc87a9-7602-4795-818c-f7151b1c9ead, use this to unset). - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:37 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:35 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: ba787a26-62db-4662-ade2-2e3061b7390d, use this to unset). - cookbook ran by andrew@buster
  • 14:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:29 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 6f0e79c6-2d91-478e-92df-0f65de48212f, use this to unset). - cookbook ran by andrew@buster
  • 14:29 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 7f577c0f-0182-498f-b198-307d51df8c3a, use this to unset). - cookbook ran by andrew@buster
  • 14:28 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:28 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:28 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:28 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: bd1bff92-bfb8-483d-a719-13fbdf3d8b21, use this to unset). - cookbook ran by andrew@buster
  • 14:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:18 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: c9cd7dbb-c5ad-4364-a8e0-afb52ef98683, use this to unset). - cookbook ran by andrew@buster
  • 14:18 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4deca8a4-f73f-4b42-b4be-acfe61421ed0, use this to unset). - cookbook ran by andrew@buster
  • 14:18 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 6d0d58b2-8450-480d-94f2-93a257f25575, use this to unset). - cookbook ran by andrew@buster
  • 14:17 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:17 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:17 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:17 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:17 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:17 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 14:13 dcaro: deleting all the leftover fullstack images (was due to max number of mysql connections reached)
  • 04:51 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:51 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:47 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:46 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 432747a2-9a15-44d4-ba8a-4e7e87eb8b76, use this to unset). - cookbook ran by andrew@buster
  • 04:45 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:45 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:45 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:45 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:45 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1c5adb8c-efcf-4036-bf22-0eef364ae399, use this to unset). - cookbook ran by andrew@buster
  • 04:44 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:44 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:41 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 26feb897-9eb0-40b6-9525-4cf44f81638d, use this to unset). - cookbook ran by andrew@buster
  • 04:41 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:40 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:40 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:39 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: cd2424d3-c9c9-4837-93b1-a6d8271e7873, use this to unset). - cookbook ran by andrew@buster
  • 04:39 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:39 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:38 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:38 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 1e85ec01-81c4-4664-b14f-b1cf6417988c, use this to unset). - cookbook ran by andrew@buster
  • 04:38 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f498aaa4-fad3-42f8-92f4-48a98c108456, use this to unset). - cookbook ran by andrew@buster
  • 04:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:37 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:37 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 0ed281f3-d9b2-4c0a-bf88-db42fe848b2f, use this to unset). - cookbook ran by andrew@buster
  • 04:35 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f417dffc-fa1b-4e65-938a-49e54b6fac6d, use this to unset). - cookbook ran by andrew@buster
  • 04:34 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:34 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:34 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:34 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:34 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:33 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 923926e0-7194-4e13-98c4-a4f703b2907b, use this to unset). - cookbook ran by andrew@buster
  • 04:32 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:32 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:29 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: ca8f729b-92ad-45ec-b5da-55987b0fc9c2, use this to unset). - cookbook ran by andrew@buster
  • 04:29 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:29 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:25 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: abe6992d-dfba-4e50-993d-de0a551a177a, use this to unset). - cookbook ran by andrew@buster
  • 04:24 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:24 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:23 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: df91107d-bf34-49db-9756-44ad06162683, use this to unset). - cookbook ran by andrew@buster
  • 04:23 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: f45ec0b1-83c9-49e0-a3a8-897d5a48414f, use this to unset). - cookbook ran by andrew@buster
  • 04:23 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:22 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:22 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:22 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:18 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 504ae503-bfad-41bd-9079-fb53a9c82a62, use this to unset). - cookbook ran by andrew@buster
  • 04:17 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:17 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:15 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 8410faf9-667d-443d-bd17-bb3ca8e7f725, use this to unset). - cookbook ran by andrew@buster
  • 04:15 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:15 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:11 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 40441a37-0dd7-44a8-960b-f74f98f53618, use this to unset). - cookbook ran by andrew@buster
  • 04:10 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:10 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:08 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 517cff2b-a643-4714-a31f-6bbe0767656a, use this to unset). - cookbook ran by andrew@buster
  • 04:08 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 50b63ad6-4d8b-4fcf-9c17-27a106b6ec52, use this to unset). - cookbook ran by andrew@buster
  • 04:07 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:07 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:07 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:07 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:06 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: d912bdf5-54c7-490a-bdbe-ba9a23f07f4b, use this to unset). - cookbook ran by andrew@buster
  • 04:06 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: e75f16d0-e42f-4d72-a3bd-be0cbc5f200a, use this to unset). - cookbook ran by andrew@buster
  • 04:06 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: 87f6c417-3703-4f73-9876-20f6767dc8e1, use this to unset). - cookbook ran by andrew@buster
  • 04:05 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:05 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:05 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:05 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:05 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:05 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:02 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: be14f869-fdb0-4c38-84ff-3d1e00d227eb, use this to unset). - cookbook ran by andrew@buster
  • 04:02 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: feb93ea7-2c64-4831-b140-dad8311dbcef, use this to unset). - cookbook ran by andrew@buster
  • 04:02 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: c65a23e3-a809-4a75-88e4-29a8b351ecdb, use this to unset). - cookbook ran by andrew@buster
  • 04:02 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:01 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:01 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:01 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:01 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:01 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:00 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:00 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:57 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:56 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8c8f64ca-a4e8-47cf-a2ce-23059109fb6a, use this to unset). - cookbook ran by andrew@buster
  • 03:55 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:55 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:54 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 9d0457e8-a861-46ce-ac35-53975a46018e, use this to unset). - cookbook ran by andrew@buster
  • 03:53 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:53 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:44 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:44 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:42 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:40 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:38 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:30 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 851d3b34-68f7-4c57-920b-2a64a9ea573d, use this to unset). - cookbook ran by andrew@buster
  • 03:29 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:29 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:17 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 1060ab82-d53a-4d48-8e15-3198cabcfc2f, use this to unset). - cookbook ran by andrew@buster
  • 03:17 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:17 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:14 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:14 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:14 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: dde1dc09-44b5-42af-ac49-c58dbbae7cb4, use this to unset). - cookbook ran by andrew@buster
  • 03:14 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:14 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:13 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:13 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: b59fa99b-8713-4e63-8c56-08bd71fdfbe3, use this to unset). - cookbook ran by andrew@buster
  • 03:13 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:13 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:07 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 3547c843-6b15-4151-b452-6f618a886b7b, use this to unset). - cookbook ran by andrew@buster
  • 03:07 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 6f5808d0-afb2-4222-ae8e-4d953d49cd63, use this to unset). - cookbook ran by andrew@buster
  • 03:06 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:06 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:06 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:06 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:06 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: da738ba3-883f-4d76-bb33-7eb551de5fc2, use this to unset). - cookbook ran by andrew@buster
  • 03:05 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:05 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:04 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:04 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:03 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f90ca4c1-78af-46bd-9262-0fa4edc2898e, use this to unset). - cookbook ran by andrew@buster
  • 03:02 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:02 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:02 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:02 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:00 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:00 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 62383663-9ae6-4a15-a2a6-62be06adbc96, use this to unset). - cookbook ran by andrew@buster
  • 02:59 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:59 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:59 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 6e0da344-49ad-451f-8b8d-ea8f853a6f89, use this to unset). - cookbook ran by andrew@buster
  • 02:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:52 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: c9ec5b7c-aa9f-4afd-a417-54c52c07ca47, use this to unset). - cookbook ran by andrew@buster
  • 02:51 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:51 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:48 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 5509c648-58f6-49fd-90e8-ac9aa66e8b04, use this to unset). - cookbook ran by andrew@buster
  • 02:48 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:48 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:46 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 02:46 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:46 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: af4ba9ad-e765-4087-86aa-9111c0821175, use this to unset). - cookbook ran by andrew@buster
  • 02:46 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:45 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:45 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: 6da98123-8838-4d2e-8659-d769bfa292fe, use this to unset). - cookbook ran by andrew@buster
  • 02:44 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:44 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:43 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 02:43 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:42 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:42 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 91860079-8fff-40db-9de7-17741d528ad6, use this to unset). - cookbook ran by andrew@buster
  • 02:41 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:41 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:39 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: a6a0f414-d305-4a09-87d2-066eec83642b, use this to unset). - cookbook ran by andrew@buster
  • 02:39 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:38 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:38 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:38 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 6000898c-4e5e-49c8-8841-3262d6da0366, use this to unset). - cookbook ran by andrew@buster
  • 02:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:34 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 24a8b50e-5c04-40e6-a732-f0232b83e240, use this to unset). - cookbook ran by andrew@buster
  • 02:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8d259aca-1803-4346-b5a4-a447673b3537, use this to unset). - cookbook ran by andrew@buster
  • 02:34 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:34 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:33 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 704aca7f-0069-4a4a-b8de-1aba7b9ca7ef, use this to unset). - cookbook ran by andrew@buster
  • 02:32 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04c0fba3-e2cb-40b4-abfd-d24165c9bd55, use this to unset). - cookbook ran by andrew@buster
  • 02:32 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:32 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 25e5e0c8-5c27-460e-9b44-1cbd1c2fb551, use this to unset). - cookbook ran by andrew@buster
  • 02:28 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:28 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:25 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5e62968a-0bb6-43b1-ae5e-e352442ef976, use this to unset). - cookbook ran by andrew@buster
  • 02:24 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:24 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:21 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 5b1a8e8f-8699-433d-b764-197862dc5cf1, use this to unset). - cookbook ran by andrew@buster
  • 02:20 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:20 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:15 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:15 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:13 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 299c466e-602b-4a8e-bb37-156553dbe426, use this to unset). - cookbook ran by andrew@buster
  • 02:13 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:13 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:12 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:12 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:03 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6a64d619-c484-411f-ad9d-89d43b063737, use this to unset). - cookbook ran by andrew@buster
  • 02:02 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:02 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:46 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-bb4aa0a75e45, use this to unset). - cookbook ran by andrew@buster
  • 00:46 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: b2f8c6a2-7224-4fb1-8e12-dff6dda02380, use this to unset). - cookbook ran by andrew@buster
  • 00:46 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:46 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:46 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:46 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:40 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:40 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: c0f62e0e-7865-4e77-a238-d707ceed53a8, use this to unset). - cookbook ran by andrew@buster
  • 00:38 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:38 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:36 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 96a368d5-232f-4187-b0bb-5923a50fff71, use this to unset). - cookbook ran by andrew@buster
  • 00:36 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:36 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:35 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 31b6bfcb-81cb-4d89-b977-190899ea2e64, use this to unset). - cookbook ran by andrew@buster
  • 00:35 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:35 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:28 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 50f763d4-36ea-4241-8e71-cc6aa20ccc9e, use this to unset). - cookbook ran by andrew@buster
  • 00:28 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: cd3d17af-ee15-4a9f-8750-66b97a46ea12, use this to unset). - cookbook ran by andrew@buster
  • 00:27 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:27 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:27 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:27 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:22 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:22 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:22 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 9b075166-d424-42c0-8e06-f74a47abb798, use this to unset). - cookbook ran by andrew@buster
  • 00:21 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:21 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:21 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:21 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster

2022-07-18

  • 23:11 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: e0aab64f-d911-4d7e-9a97-c59d9868ceea, use this to unset). - cookbook ran by andrew@buster
  • 23:11 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 447d9c7a-a00e-41db-93ed-6d2bfc66e5df, use this to unset). - cookbook ran by andrew@buster
  • 23:11 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:11 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:11 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:11 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:01 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 46c2ddf2-50c3-4824-ba72-d7fb3fa4c5e0, use this to unset). - cookbook ran by andrew@buster
  • 23:01 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 6facdf4f-7bb8-42cd-8ac8-b4412fc4cf70, use this to unset). - cookbook ran by andrew@buster
  • 23:00 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:00 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:00 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:00 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:52 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 23190573-fc0d-4872-8343-e35b84132028, use this to unset). - cookbook ran by andrew@buster
  • 22:51 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:51 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:51 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 990313c4-5fa6-4de7-a8bf-b361f0c79e90, use this to unset). - cookbook ran by andrew@buster
  • 22:50 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:50 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:48 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 22:48 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 22:42 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 59367d39-bd2f-47fd-becd-255bf823ff12, use this to unset). - cookbook ran by andrew@buster
  • 22:42 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:42 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:40 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 38602815-d187-4455-9676-59607505bba7, use this to unset). - cookbook ran by andrew@buster
  • 22:39 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 00f21e1b-d014-4bc5-995f-e25201fc77c4, use this to unset). - cookbook ran by andrew@buster
  • 22:39 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:39 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:39 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:39 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:31 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: d05d07b2-8c60-46de-a89e-76987901901b, use this to unset). - cookbook ran by andrew@buster
  • 22:31 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 47f6b9e6-ceab-4fb6-8f76-0de4341d0841, use this to unset). - cookbook ran by andrew@buster
  • 22:31 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:31 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:30 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:30 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:30 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 599042a5-da25-4985-b502-328fe40a72d5, use this to unset). - cookbook ran by andrew@buster
  • 22:29 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:29 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:29 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: a554d107-c14d-41f8-b180-093e2cf73e72, use this to unset). - cookbook ran by andrew@buster
  • 22:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9fef70c7-6f7d-4b33-a9bc-8c39630dc182, use this to unset). - cookbook ran by andrew@buster
  • 22:26 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:02 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 3744c0fa-084d-4198-98ca-66c7a5210f41, use this to unset). - cookbook ran by andrew@buster
  • 22:02 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 01caf1e1-df64-474a-94d5-f37751e23d62, use this to unset). - cookbook ran by andrew@buster
  • 22:01 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:01 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:01 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:01 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:59 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 3018c2fc-9cd5-45c8-a160-d5ad5728c8d5, use this to unset). - cookbook ran by andrew@buster
  • 21:59 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:59 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: d760f916-32ea-4194-8974-f36064a9866c, use this to unset). - cookbook ran by andrew@buster
  • 21:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:00 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: cb2a38f3-e9ca-4bf6-989c-a27be2b2d88c, use this to unset). - cookbook ran by andrew@buster
  • 20:59 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:59 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:58 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 93fa9477-54b8-46e7-8508-a2d492528d1f, use this to unset). - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:52 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f6cf5b29-ce8b-426e-af1c-cab74e0c4a3a, use this to unset). - cookbook ran by andrew@buster
  • 20:52 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: fe26e612-d76e-4120-b60a-4f300add224f, use this to unset). - cookbook ran by andrew@buster
  • 20:52 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 2abd94be-b6d8-4042-a9ce-998e41d932f4, use this to unset). - cookbook ran by andrew@buster
  • 20:51 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:51 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:51 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:51 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:51 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:51 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 10973c71-30d8-44bf-a310-d0c1af76d399, use this to unset). - cookbook ran by andrew@buster
  • 20:28 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: f8f1b214-ec76-4157-a8e6-6374e5d14608, use this to unset). - cookbook ran by andrew@buster
  • 20:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:27 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:27 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:11 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 95dae613-2b04-472e-a8f9-e71c3fbb8d0e, use this to unset). - cookbook ran by andrew@buster
  • 20:10 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:10 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: c804f4db-ea63-44e9-aff8-fea54c238345, use this to unset). - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 597a901d-4624-43de-b986-c4c6c2fcf80c, use this to unset). - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:03 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: fdceed0d-6fa0-4806-8fb6-3b321a5a1623, use this to unset). - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:01 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 20:01 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:57 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:37 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: b0904345-9aa3-4202-b992-78644141517c, use this to unset). - cookbook ran by andrew@buster
  • 19:36 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:36 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:31 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:31 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:30 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster

2022-07-13

  • 14:48 bd808: Added Tavvi as member of #acl*wmcs-team

2022-07-12

  • 12:04 wm-bot2: Ceph cluster at {self.deployment} set out of maintenance. - cookbook ran by dcaro@vulcanus
  • 12:03 wm-bot2: Set the ceph cluster for codfw1dev in maintenance, alert silence ids: db32805f-a033-4e0e-8fc7-ed0c2e9f8be1,f4e698f0-4b51-4b07-9ba8-0b296ca1c4fb,39ad5325-44ed-44d1-bba5-91a85c7a8401,a584a3c6-9dae-41e9-ac82-a91cd35af8fb - cookbook ran by dcaro@vulcanus
  • 08:37 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2005-dev', 'cloudnet2006-dev'] - cookbook ran by dcaro@vulcanus
  • 08:37 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:31 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2005-dev,cloudnet2006-dev - cookbook ran by dcaro@vulcanus

2022-07-08

  • 15:57 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 15:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

2022-07-07

  • 07:17 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet (T312509) - cookbook ran by dcaro@vulcanus
  • 07:12 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet (T312509) - cookbook ran by dcaro@vulcanus

2022-07-06

  • 17:50 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

2022-07-04

  • 13:27 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:27 wm-bot2: Rebooted cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:23 wm-bot2: Rebooting cloudgw host cloudgw1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:23 wm-bot2: Rebooting all the cloudgw nodes from the eqiad1 deployment: cloudgw1001.eqiad.wmnet,cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:13 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
  • 13:12 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:09 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:08 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:05 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:05 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 12:56 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 12:52 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:56 wm-bot2: Finished rebooting the cloudgw nodes ['cloudgw2001-dev.codfw.wmnet', 'cloudgw2002-dev.codfw.wmnet', 'cloudgw2003-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
  • 10:56 wm-bot2: Rebooted cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:51 wm-bot2: Rebooting cloudgw host cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:51 wm-bot2: Rebooted cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:47 wm-bot2: Rebooting cloudgw host cloudgw2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:47 wm-bot2: Rebooted cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:42 wm-bot2: Rebooting cloudgw host cloudgw2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:42 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:41 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 10:40 wm-bot2: Rebooting all the cloudgw nodes cloudgw2001-dev.codfw.wmnet,cloudgw2002-dev.codfw.wmnet,cloudgw2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 09:07 wm-bot2: Rebooting cloudnet host cloudnet1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 09:06 wm-bot2: Rebooted cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 09:00 wm-bot2: Rebooting cloudnet host cloudnet1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 09:00 wm-bot2: Rebooting all the cloudnet nodes cloudnet1003,cloudnet1004 - cookbook ran by dcaro@vulcanus
  • 08:59 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 08:59 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:53 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:53 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:47 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:47 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 08:32 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 08:32 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:25 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:25 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 08:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 07:55 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 07:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 07:44 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 07:44 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:40 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:36 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:36 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 07:09 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:05 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 07:04 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

2022-07-03

  • 21:27 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

2022-07-02

  • 11:05 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

2022-07-01

  • 15:35 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 15:35 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:30 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:29 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:25 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:25 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 15:13 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 15:13 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:08 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:07 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:03 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 15:03 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 14:42 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 14:42 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:37 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:36 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:32 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:31 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 14:23 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:19 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:19 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 14:19 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus
  • 14:09 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:06 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:02 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:58 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:58 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 13:55 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:54 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 13:52 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:51 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 13:37 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus
  • 13:34 wm-bot2: Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:31 wm-bot2: Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 13:30 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

2022-06-30

  • 18:17 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 18:13 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:45 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 07:40 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 06:52 wm-bot2: Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

2022-06-29

  • 08:45 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 08:41 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

2022-06-28

  • 13:03 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor T301949

2022-06-22

  • 16:48 taavi: restart designate-*.service on both cloudservices nodes
  • 16:44 taavi: restart nova-conductor on all the cloudcontrol nodes
  • 12:50 andrewbogott: rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability

2022-06-21

  • 04:48 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
  • 04:02 andrewbogott: restarting rabbitmq on cloudcontrol100x (one at a time)

2022-06-17

  • 17:53 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. T310664
  • 06:15 taavi: restart neutron-linuxbridge-agent on cloudvirt1046

2022-06-15

  • 11:34 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

2022-06-14

  • 16:26 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl
  • 14:38 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudceph
  • 12:55 wm-bot2: OSDs (['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev']) upgraded successfully B-) (T309786) - cookbook ran by dcaro@vulcanus
  • 12:44 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T309786) - cookbook ran by dcaro@vulcanus

2022-06-13

  • 11:14 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus
  • 11:08 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus
  • 11:07 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus
  • 11:05 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus
  • 11:04 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus
  • 11:03 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus
  • 09:15 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

2022-06-06

  • 13:21 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

2022-06-02

  • 07:51 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 T309732
  • 00:37 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

2022-06-01

  • 17:11 andrewbogott: restarting designate services in cloudservices1xxx hosts
  • 16:37 taavi: root@cloudcontrol1005:~# cinder reset-state --state available 491066a4-16a9-4ce4-a9f6-182660616d77 for T309659

2022-05-30

  • 17:43 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

2022-05-29

  • 14:55 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage
  • 14:15 andrewbogott: restarting rabbitmq on all eqiad1 cloudcontrol nodes (one at a time)

2022-05-25

  • 20:01 balloons: clean up cinder backup a bit, restart service due to network outage
  • 20:01 balloons: cloudvirt1029 restarted nova due to network outage

2022-05-19

  • 15:21 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

2022-05-18

  • 15:42 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

2022-05-14

  • 11:33 taavi: deleted projects 'ores' and 'ores-staging' T308102

2022-05-13

  • 06:20 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 06:20 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 06:16 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:16 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 06:15 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:15 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:11 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 06:11 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 06:10 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 06:10 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:09 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:07 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:06 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 06:06 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 06:05 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:51 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:50 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:50 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:49 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:49 wm-bot2: Safe reboot of 'cloudvirt1043.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 05:49 wm-bot2: Unset cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:49 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:49 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:47 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 05:47 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:45 wm-bot2: Drained 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:45 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:45 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:44 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:44 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:42 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:42 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:42 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:41 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:40 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:40 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:38 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:37 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:37 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:30 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:29 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:29 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:19 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:18 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:18 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:18 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:18 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:18 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:12 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 05:12 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:08 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:02 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:02 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 05:02 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:02 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:02 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 05:01 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:52 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:51 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:51 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:48 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:48 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:44 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:31 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:30 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:30 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:30 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:30 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:27 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:26 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:26 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:26 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:26 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:25 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:25 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:24 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:23 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:23 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:23 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:22 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:22 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:21 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 04:21 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:18 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:16 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:16 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:16 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:16 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:15 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:15 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:37 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:36 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:36 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:34 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:34 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:27 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:27 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:26 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:26 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:26 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:25 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:25 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:22 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:55 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:55 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:54 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:54 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:54 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:54 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:05 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:05 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:05 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:04 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:23 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 01:23 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 01:19 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:01 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 01:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 01:00 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:00 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:25 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 00:25 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:21 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:20 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:19 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:19 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 00:11 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 00:11 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:07 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster

2022-05-12

  • 23:55 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 23:55 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 23:54 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:54 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:54 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 23:54 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:23 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 22:23 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 22:20 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 22:17 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 22:17 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 22:13 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:57 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:56 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:56 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:55 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:55 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:54 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:54 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:53 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:52 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:52 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:51 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:44 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:44 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:42 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:36 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:36 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:33 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:33 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:32 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:32 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:32 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:31 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:31 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:31 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:31 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:30 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:29 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:29 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:29 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:28 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:28 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:28 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:22 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:22 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:21 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:21 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:18 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:18 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:18 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:17 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:17 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:11 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 21:11 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:08 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:08 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:07 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:07 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:05 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:04 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:04 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:00 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:59 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:59 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:59 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 20:59 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:57 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:55 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:55 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:54 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:54 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:54 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:53 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:52 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:52 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:50 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:49 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:49 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:49 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:48 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 20:48 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:44 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:44 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:43 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:43 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:34 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 20:31 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:28 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:28 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:27 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:27 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:23 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:22 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:22 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:11 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:10 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:10 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:07 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:07 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:06 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 20:05 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:02 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:01 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:00 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:00 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:58 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:57 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:57 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:36 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:35 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:35 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:06 andrewbogott: stopping nfs-server on labstore1004 in preparation for reboot
  • 04:12 andrewbogott: rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying

2022-05-11

  • 18:48 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 18:48 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:48 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:39 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 18:38 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:38 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:04 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 18:03 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 18:03 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 08:56 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 08:52 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 07:53 dcaro: test
  • 04:28 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 04:27 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 04:27 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:44 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:43 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:43 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:42 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 03:42 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:39 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:23 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:22 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:22 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:09 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:08 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:08 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:04 andrewbogott: reset and recreated the rabbitmq cluster in eqiad1 to get around some broken queues.
  • 03:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 03:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 03:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 02:25 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 02:25 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster

2022-05-10

  • 21:43 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:40 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:40 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:35 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 21:32 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 21:32 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:05 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 20:02 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:01 wm-bot: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:00 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 20:00 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:57 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:57 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:55 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:55 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:47 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:46 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:46 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:45 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:45 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:44 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:44 wm-bot: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:40 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:39 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:39 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:37 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:36 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:36 wm-bot: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:33 wm-bot: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • 19:33 wm-bot: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:29 wm-bot: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:06 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 19:05 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 19:05 wm-bot: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 15:41 andrewbogott: rebooting cloud*-dev for T307668
  • 13:59 taavi: manually attached User:Dreamy Jazz to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)

2022-05-07

  • 01:33 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:32 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 01:30 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster
  • 01:21 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 01:18 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

2022-05-03

  • 20:38 andrewbogott: upgrading clouddb2001-dev in place
  • 18:18 taavi: updated 'puppet-enc' endpoints on the keystone catalog to use https and port 443

2022-05-02

  • 16:56 dcaro: rebooting cloudmetrics1001

2022-04-29

2022-04-27

  • 14:51 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud
  • 14:50 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:46 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:46 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:41 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:41 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:35 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:35 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:31 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:31 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:27 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:27 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:23 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:23 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:13 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:13 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:09 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:09 wm-bot: Finished rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:05 wm-bot: Rebooting node cloudcephosd1016.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:05 wm-bot: Finished rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:01 wm-bot: Rebooting node cloudcephosd1015.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:01 wm-bot: Finished rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:57 wm-bot: Rebooting node cloudcephosd1014.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:57 wm-bot: Finished rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:44 wm-bot: Rebooting node cloudcephosd1013.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:43 wm-bot: Finished rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:39 wm-bot: Rebooting node cloudcephosd1012.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:39 wm-bot: Finished rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:35 wm-bot: Rebooting node cloudcephosd1011.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:35 wm-bot: Finished rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:31 wm-bot: Rebooting node cloudcephosd1010.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:31 wm-bot: Finished rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:26 wm-bot: Rebooting node cloudcephosd1009.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:26 wm-bot: Finished rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:14 wm-bot: Rebooting node cloudcephosd1008.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:14 wm-bot: Finished rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:10 wm-bot: Rebooting node cloudcephosd1007.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:10 wm-bot: Finished rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:05 wm-bot: Rebooting node cloudcephosd1006.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:05 wm-bot: Finished rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot: Rebooting node cloudcephosd1005.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot: Finished rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:57 wm-bot: Rebooting node cloudcephosd1004.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:57 wm-bot: Finished rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:53 wm-bot: Rebooting node cloudcephosd1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:53 wm-bot: Finished rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:50 wm-bot: Rebooting node cloudcephosd1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:50 wm-bot: Finished rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:46 wm-bot: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:46 wm-bot: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024 - cookbo
  • 12:15 wm-bot: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] - cookbook ran by dcaro@vulcanus
  • 12:15 wm-bot: Finished rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:12 wm-bot: Rebooting node cloudcephmon1003.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:12 wm-bot: Finished rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:09 wm-bot: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:09 wm-bot: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:07 wm-bot: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 12:07 wm-bot: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by dcaro@vulcanus
  • 12:05 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] - cookbook ran by dcaro@vulcanus
  • 12:05 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 12:02 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 12:02 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:59 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:59 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:56 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:56 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev - cookbook ran by dcaro@vulcanus
  • 11:55 wm-bot: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by dcaro@vulcanus
  • 11:55 wm-bot: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:52 wm-bot: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:52 wm-bot: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:47 wm-bot: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:47 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:43 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 11:43 wm-bot: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by dcaro@vulcanus

2022-04-26

2022-04-22

  • 10:33 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

2022-04-21

  • 05:38 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

2022-04-19

  • 15:29 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

2022-04-18

  • 15:23 andrewbogott: reimaging cloudvirt1020, leaving VMs in place
  • 13:40 andrewbogott: shutting down many codfdfw1dev servers (including network infra!) for T305469

2022-04-14

  • 20:14 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

2022-04-13

  • 22:01 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

2022-04-11

  • 15:59 taavi: created cloudinfra.wmcloud.org zone

2022-04-09

  • 19:55 andrewbogott: reimaging cloudbackup1001-dev to bullseye
  • 19:37 taavi: add 'puppet-enc' service & endpoint to keystone T274666
  • 19:25 andrewbogott: reimaging cloudbackup1002-dev to bullseye

2022-04-07

  • 12:51 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T305631) - cookbook ran by arturo@nostromo

2022-04-06

  • 09:12 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) (T305157)
  • 08:45 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudvirt2003-dev (T305157)
  • 08:42 arturo: [codfw1dev] trying with python3-eventlet 0.30.2-5 installed by hand on cloudcontrol servers (T305157)
  • 08:24 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudvirt2003-dev (T305157)
  • 08:20 arturo: [codfw1dev] trying with python3-dnspython 2.2.0-2 installed by hand on cloudcontrol servers (T305157)

2022-03-30

  • 11:20 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - T285461

2022-03-29

  • 10:02 dcaro: restarting keystone (T304918)

2022-03-23

  • 22:53 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 22:38 wm-bot: Drained 'cloudvirt1044.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 22:12 wm-bot: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 22:12 wm-bot: Draining 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 22:08 wm-bot: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 22:07 wm-bot: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 22:06 wm-bot: Draining 'cloudvirt1044.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 22:06 wm-bot: Draining 'cloudvirt1043.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 21:54 wm-bot: Drained 'cloudvirt1042.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 21:19 wm-bot: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 21:19 wm-bot: Draining 'cloudvirt1042.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 21:12 wm-bot: Drained 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 21:12 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 21:09 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 21:07 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 21:04 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:55 wm-bot: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:54 wm-bot: Draining 'cloudvirt1041.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:30 wm-bot: Drained 'cloudvirt1039.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:15 wm-bot: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:15 wm-bot: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:14 wm-bot: Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:14 wm-bot: Draining 'cloudvirt1039.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:44 wm-bot: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:43 wm-bot: Draining 'cloudvirt1038.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:19 wm-bot: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:18 wm-bot: Draining 'cloudvirt1037.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:13 wm-bot: Drained 'cloudvirt1036.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:02 wm-bot2: Testing wm-bot relay to #wikimedia-cloud-feed
  • 17:55 wm-bot: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 17:54 wm-bot: Draining 'cloudvirt1036.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 17:04 wm-bot: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 17:03 wm-bot: Draining 'cloudvirt1035.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 17:03 wm-bot: Drained 'cloudvirt1034.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:51 wm-bot: Drained 'cloudvirt1033.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:37 wm-bot: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:37 wm-bot: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:36 wm-bot: Draining 'cloudvirt1034.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:36 wm-bot: Draining 'cloudvirt1033.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 15:01 wm-bot: Drained 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 15:00 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 14:57 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 14:44 wm-bot: Drained 'cloudvirt1031.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 14:35 wm-bot: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 14:34 wm-bot: Draining 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 14:32 wm-bot: Drained 'cloudvirt1030.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 14:20 wm-bot: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 14:19 wm-bot: Draining 'cloudvirt1031.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 14:18 wm-bot: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 14:17 wm-bot: Draining 'cloudvirt1030.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 13:54 taavi: restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change
  • 13:43 wm-bot: Drained 'cloudvirt1029.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 13:23 wm-bot: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 13:22 wm-bot: Draining 'cloudvirt1029.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

2022-03-22

  • 22:59 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 22:58 wm-bot: Draining 'cloudvirt1027.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

2022-03-17

  • 01:09 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 00:53 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 00:52 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 00:52 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

2022-03-15

  • 20:58 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:36 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:36 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:36 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 13:14 wm-bot: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
  • 13:14 wm-bot: Unsetting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
  • 10:32 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo
  • 10:30 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by arturo@nostromo

2022-03-14

  • 21:24 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:59 wm-bot: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:58 wm-bot: Setting cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:58 wm-bot: Draining 'cloudvirt1025.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 20:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 20:02 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 19:59 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 19:59 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 19:16 wm-bot: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 19:15 wm-bot: Setting cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 19:15 wm-bot: Draining 'cloudvirt1024.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 19:13 wm-bot: Drained 'cloudvirt1023.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:56 wm-bot: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:55 wm-bot: Setting cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:55 wm-bot: Draining 'cloudvirt1023.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:53 wm-bot: Drained 'cloudvirt1022.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:52 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:51 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:51 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:50 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:48 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:48 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:48 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 11:48 dcaro: rebased cookbooks on latest master, make sure you pull before sending new patches

2022-03-08

  • 18:29 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:29 wm-bot: Setting cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:29 wm-bot: Draining 'cloudvirt1022.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:23 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:21 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:21 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 18:18 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:17 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 18:17 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 17:28 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 17:27 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 17:27 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 17:18 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 17:15 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 17:15 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:48 wm-bot: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:47 wm-bot: Setting cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:47 wm-bot: Draining 'cloudvirt1017.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:36 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 16:08 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:07 wm-bot: Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster
  • 16:07 wm-bot: Draining 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster
  • 13:11 arturo: [codfw1dev] rebooting cloudservices servers for T303179
  • 13:07 arturo: [codfw1dev] rebooting cloudvirt servers for T303179
  • 13:06 arturo: [codfw1dev] rebooting cloudnet servers for T303179
  • 12:55 arturo: [codfw1dev] rebooting cloudcontrol servers for T303179

2022-03-03

  • 08:49 taavi: deploying cloudmetrics grafana to grafana 8, T282863

2022-03-02

2022-02-28

  • 15:30 dcaro: cleaning up leftover snapshots from failed backups of the maps volume (T302720)

2022-02-24

  • 17:04 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'
  • 15:42 dcaro: stopping and starting mariadb on cloudcontrol1003 (T302146)
  • 10:37 arturo: [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install (T302482)

2022-02-23

  • 20:39 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account T295246
  • 17:40 andrewbogott: restarting lots of openstack services to try to clear up the mess that is T236101
  • 12:13 arturo: cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done (T302382)
  • 10:14 arturo: cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet
  • 10:05 dcaro: Deleting stuck novafullstack servers, to let the service create new ones (T302369)
  • 09:56 arturo: neutron agent-delete bad663b3-fd25-4393-a546-4b1b4bdec4db (Linux bridge agent | cloudvirtan1001)
  • 09:56 arturo: neutron agent-delete 1071c198-ed57-4b5a-9439-30e66a31aa69 (Linux bridge agent | cloudvirtan1005)
  • 09:55 arturo: neutron agent-delete 2eeef198-8af7-4e5d-bd73-e14a2a8d2404 (Linux bridge agent | cloudvirtan1004)
  • 09:55 arturo: neutron agent-delete afe173eb-35ba-444a-9960-899629786d2f (Linux bridge agent | cloudvirtan1003)
  • 09:54 arturo: neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-92bfbe8d1a56 (Linux bridge agent | cloudvirtan1002)
  • 09:39 dcaro: restarting neutron-api cloudcontrol1003 to see if the agent status update starts working (T302369)
  • 09:38 dcaro: restarting neutron-dhcp-agent on cloudnet1003 (T302369)

2022-02-22

  • 22:10 andrewbogott: raising project 'maps' quota by two tb -- T300160
  • 09:24 arturo: restarting mariadb @ cloudcontrol1003 (T302146)
  • 09:13 arturo: restarting mariadb @ cloudcontrol1004 (T302146)

2022-02-18

  • 21:57 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend. Everything there should be stable and fine save rabbit which needs an upgrade.
  • 21:30 andrewbogott: rebooting cloudcontrol1003 because rabbit is freaking out
  • 17:25 andrewbogott: in-place upgrade of cloudcontrol1004 to bullseye -- T281276
  • 12:34 arturo: manually install prometheus-openstack-exporter on cloudcontrol1005 (T302050)

2022-02-17

  • 23:02 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 T281276

2022-02-15

  • 14:15 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account T295246

2022-02-04

  • 10:12 arturo: restart backup_vms service in cloudvirt1024 (T300956)

2022-02-03

  • 08:21 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing "performing a change every puppet run" alert)

2022-02-02

  • 02:36 andrewbogott: restarting mariadb on cloudcontrol1004

2022-01-31

  • 10:15 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

2022-01-29

  • 18:24 taavi: delete 2 puppet prefixes in a weird state T299750

2022-01-27

  • 13:24 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service (T300138)

2022-01-26

  • 19:09 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004
  • 18:57 andrewbogott: restarting mariadb on cloudcontrol1004

2022-01-25

2022-01-19

  • 16:38 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

2022-01-05

2022-01-04

  • 12:44 dcaro: increasing the size_limit for labs ldap servers

2021-12-26

  • 16:55 majavah: run attachLdapUser.php on wikitech for developer account "Karthiksripal"

2021-12-24

  • 22:51 majavah: ran the wikireplica dns script on s5 T298303

2021-12-23

  • 21:42 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

2021-12-21

  • 10:39 arturo: dropped egress NAT exceptions for WMF apt repos, T298042

2021-12-15

  • 12:44 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed (T297454)

2021-12-14

  • 10:26 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/<canary_image_id>) to the root disk on cloudvirt-wdqs1001 to temporary free some space (T297454)

2021-12-13

  • 18:08 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
  • 17:50 wm-bot: Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 17:49 wm-bot: Setting cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 17:49 wm-bot: Draining 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse
  • 17:44 wm-bot: Drained 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
  • 17:30 wm-bot: Set cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 17:30 wm-bot: Setting cloudvirt 'cloudvirt1013.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 17:30 wm-bot: Draining 'cloudvirt1013.eqiad.wmnet'. - cookbook ran by michael@mouse
  • 17:13 wm-bot: Drained 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
  • 16:50 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 16:47 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 16:47 wm-bot: Draining 'cloudvirt1012.eqiad.wmnet'. - cookbook ran by michael@mouse
  • 16:44 wm-bot: Set cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse
  • 16:43 wm-bot: Setting cloudvirt 'cloudvirt1012.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse

2021-12-03

  • 18:56 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
  • 10:49 majavah: deleting dbbackups-dashboard project T296992

2021-12-02

  • 01:17 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster
  • 00:56 wm-bot: Set cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:56 wm-bot: Setting cloudvirt 'cloudvirt1028.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:56 wm-bot: Draining 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster
  • 00:50 wm-bot: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:50 wm-bot: Setting cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:50 wm-bot: Draining 'cloudvirt1026.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster
  • 00:28 wm-bot: Drained 'cloudvirt1021.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster
  • 00:03 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:02 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 00:02 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster

2021-12-01

  • 23:59 wm-bot: Setting cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
  • 23:59 wm-bot: Draining 'cloudvirt1021.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster
  • 23:54 andrewbogott: *correction* adding spare cloudvirts 1044 and 1045 to the 'ceph' pool in order to make space for future juggling around T296790 and T296792
  • 23:53 andrewbogott: adding spare cloudvirts 1044 and 1055 to the 'ceph' pool in order to make space for future juggling around T296790 and T296792

2021-11-28

  • 17:48 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for T296592. It will need to be moved back after the raid is rebuilt.

2021-11-21

  • 07:19 dcaro_away: restarting designate-sink with some extra logs in it (T296144)

2021-11-17

  • 15:48 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
  • 15:39 andrewbogott: sudo cumin "cloud*" 'apt-get update -y --allow-releaseinfo-change'
  • 15:26 andrewbogott: updated mariadb packages on codfw1dev cloudcontrols to 1:10.3.31-0+deb10u1

2021-11-12

  • 13:31 arturo: restarting glance-api services to make sure they work with new ceph auth creds (T293752)

2021-11-08

2021-11-05

  • 11:18 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance
  • 11:17 wm-bot: Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 11:15 wm-bot: Finished rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
  • 11:12 wm-bot: Rebooting node cloudcephosd1024.eqiad.wmnet - cookbook ran by arturo@endurance
  • 11:12 wm-bot: Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 11:12 wm-bot: Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

2021-11-04

  • 16:39 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance
  • 16:39 wm-bot: Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 16:37 wm-bot: Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
  • 16:34 wm-bot: Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance
  • 16:33 wm-bot: Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 16:33 wm-bot: Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance
  • 16:17 wm-bot: Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance
  • 16:17 wm-bot: Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 16:16 wm-bot: Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
  • 16:13 wm-bot: Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance
  • 16:12 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 16:12 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance
  • 16:00 wm-bot: Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 16:00 wm-bot: Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance
  • 11:26 wm-bot: Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance
  • 11:26 wm-bot: Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 11:23 wm-bot: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
  • 11:20 wm-bot: Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance
  • 11:19 wm-bot: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance
  • 11:19 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance
  • 11:16 wm-bot: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

2021-11-03

  • 17:22 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev (T294956)
  • 11:45 arturo: [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev (T294853, T291813)

2021-11-02

  • 10:54 arturo: rebooting cloudnet1004/1003 for T291813
  • 10:43 arturo: [codfw1dev] rebooting cloudgw200[12]-dev for T291813

2021-10-24

2021-10-21

  • 10:19 arturo: drop firewall exception on core routers for wiki replicas legacy setup (T293897)
  • 10:12 arturo: drop NAT exception for wiki replicas legacy setup (T293897)

2021-10-20

  • 21:06 andrewbogott: creating cloudinfra-nfs project T293936

2021-10-18

  • 19:21 andrewbogott: also ticked the 'admin' box on wikitech for majavah T292827
  • 18:58 andrewbogott: granting majavah 'admin' role in the 'admin' project and also in the default domain. T292827

2021-10-14

  • 12:28 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB (T292546)

2021-10-13

  • 10:46 arturo: updating python3-neutron across the fleet (T292936)

2021-10-12

  • 09:06 dcaro: upgrading eqiad cloudnet hosts neutron packages (T292936)
  • 08:57 dcaro: upgrading codfw cloudnet hosts neutron packages (T292936)

2021-10-05

  • 09:39 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257

2021-09-30

  • 14:50 andrewbogott: sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202
  • 14:43 andrewbogott: ran sudo cumin --force --timeout 500 -o json "A:all" "ps -ef | grep nslcd && service nslcd restart" to get nslcd happy again T292202

2021-09-29

  • 09:41 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257)

2021-09-28

  • 16:23 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly T291963
  • 16:21 bstorm: powering on clouddb1020 via remote console T291963
  • 15:58 bstorm: depooled clouddb1020 for repair T291961
  • 12:40 dcaro: Merged change on sssd for bullseye cloud hosts (T291585)
  • 11:30 arturo: [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (T291257)

2021-09-27

  • 10:07 arturo: cloudcontrol1004 apparently healthy T291446
  • 09:25 arturo: rebooting cloudcontrol1004 for T291446

2021-09-24

  • 13:02 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
  • 13:00 arturo: [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict

2021-09-21

  • 12:13 arturo: [codfw1dev] trying to create a manila service image (T291257)
  • 11:45 arturo: [codfw1dev] created rabbitmq user (T291257)
  • 11:32 arturo: [codfw1dev] populated manila DB & created service endpoints (T291257)
  • 11:06 arturo: [codfw1dev] give manila user admin role @ manila project (T291257)
  • 11:06 arturo: [codfw1dev] created manila project (T291257)
  • 10:57 arturo: [codfw1dev] created manila user @ labtestwikitech (T291257)
  • 10:49 arturo: [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) T291257

2021-09-20

  • 23:08 bstorm: ran `echo check > /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
  • 22:48 andrewbogott: stopped puppet & mariadb on cloudcontrol1004; it was flapping
  • 22:44 andrewbogott: sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there
  • 21:57 andrewbogott: moving cloudvirt1043 into the 'nfs' aggregate for T291405

2021-09-17

  • 11:35 arturo: [codfw1dev] install manila on cloudcontrol2001-dev (T291257)

2021-09-16

  • 15:56 bstorm: removing downtime for labstore1005 so we'll know if it has another issue T290318

2021-09-09

  • 22:03 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working T290630
  • 03:15 bstorm: resetting swap on clouddb1017 T290630
  • 03:08 andrewbogott: stopping maintain-dbusers on labstore1004 for help diagnosing T290630

2021-09-03

  • 15:34 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 T290318
  • 15:24 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2002 T290318
  • 15:20 bstorm: stopping puppet and disabling backup syncs to labstore1005 on cloudbackup2001 T290318

2021-08-30

  • 16:16 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
  • 16:16 wm-bot: Added OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
  • 16:13 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
  • 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 16:10 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • 16:07 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • 16:07 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
  • 16:07 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

2021-08-27

  • 18:57 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

2021-08-25

  • 14:45 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • 14:42 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • 14:42 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
  • 14:42 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 14:41 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster

2021-08-19

  • 17:39 bstorm: restarting glance image backup to try and clear the page

2021-08-18

  • 16:21 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • 16:21 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) - cookbook ran by andrew@buster
  • 16:21 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 16:17 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 16:16 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 16:15 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 16:13 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster - cookbook ran by andrew@buster
  • 14:47 andrewbogott: adding clouvirt1038 to the ceph aggregate, removing from the maintenance aggregate T276922

2021-08-17

  • 15:11 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- T287838

2021-08-11

  • 13:51 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:48 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 13:47 wm-bot: Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) (T285858) - cookbook ran by dcaro@vulcanus
  • 13:47 wm-bot: Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

2021-08-10

  • 15:15 andrewbogott: restarting all designate services in eqiad1
  • 15:04 andrewbogott: restarting designate-sink in eqiad1; it's complaining about rabbit but I don't want to restart rabbit yet

2021-08-05

  • 09:37 dcaro: Taking one osd daemon down ot codfw cluster (T288203)

2021-08-04

  • 19:20 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

2021-08-03

  • 17:40 bstorm: rerunning the glance backup script after failure

2021-07-31

  • 00:10 andrewbogott: "systemctl reset-failed cloud-init.service" on all VMs for T287309
  • 00:08 andrewbogott: "systemctl reset-failed cloud-final.service" on all VMs for T287309

2021-07-27

  • 21:32 andrewbogott: putting cloudvirt1012 back into service T286748
  • 20:52 andrewbogott: draining VMs off of cloudvirt1012 so we can replace the battery for T286748
  • 15:15 andrewbogott: "rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list" cloud-wide

2021-07-23

  • 15:22 bstorm: update wikireplicas-dns for s7 fix for web replicas

2021-07-20

  • 17:07 andrewbogott: reloading haproxy on dbproxy1018 for T286598
  • 15:45 arturo: failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417
  • 00:10 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater

2021-07-19

  • 22:05 bstorm: set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 T286599
  • 20:40 andrewbogott: reloading haproxy on dbproxy1018 for T286598
  • 13:50 andrewbogott: upgrading mariadb to 10.3.29 on all cloudcontrols

2021-07-16

  • 09:55 dcaro: checking HP raid issues on coludvirt1012 (T286766)

2021-07-14

  • 21:08 andrewbogott: restarting lots of openstack services while trying to resolve T286675
  • 12:17 dcaro: doing ceph outage tests on codfw1 (fyi)

2021-07-13

  • 10:57 dcaro: enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128

2021-07-02

  • 10:12 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus
  • 10:12 wm-bot: Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus
  • 10:12 wm-bot: Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus
  • 10:10 wm-bot: Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 10:07 wm-bot: Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 10:07 wm-bot: Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus
  • 10:07 wm-bot: Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus
  • 10:05 wm-bot: Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 10:02 wm-bot: Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 10:02 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus
  • 10:01 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 09:13 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus
  • 09:13 wm-bot: Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

2021-07-01

  • 16:27 bstorm: failed over cloudstore1009 to cloudstore1008 T224747
  • 16:18 bstorm: downtimed cloudstore1008 and cloudstore1009 to fail over T224747
  • 14:25 wm-bot: Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) (T285858) - cookbook ran by dcaro@vulcanus
  • 14:25 wm-bot: Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus
  • 14:24 wm-bot: Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:21 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:20 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus
  • 14:20 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 14:18 wm-bot: Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • 14:17 wm-bot: Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus
  • 14:17 wm-bot: Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 11:16 wm-bot: Added new OSD node cloudcephosd1016.eqiad.wmnet (T285858) - cookbook ran by dcaro@vulcanus
  • 11:13 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 10:58 dcaro: rebooting cloudcephosd1016 (T285858)
  • 10:47 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 10:44 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 10:42 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 10:41 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus
  • 10:40 wm-bot: Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

2021-06-30

  • 21:48 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration

2021-06-25

  • 15:28 andrewbogott: restarting openstack services on cloudcontrol1005
  • 09:16 arturo: icinga downtime cloudcontrols for 2h
  • 08:20 dcaro: restarting rabbitmq on cloudcontrol100{3,4}

2021-06-21

  • 13:54 dcaro: puppet fix merged and deployed, servers are back to normal
  • 13:20 dcaro: merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer)

2021-06-20

  • 22:21 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster

2021-06-15

  • 01:18 bstorm: running a modified version of the prometheus dir size cron in screen T284964

2021-06-14

  • 10:13 dcaro: setting ssd to debug mode on tools-sgeexec-0917 (T284130)

2021-06-10

  • 10:58 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 10:58 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:55 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:55 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:52 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:52 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:49 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:49 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus
  • 10:48 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 10:48 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:45 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:45 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:42 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:42 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:39 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 10:39 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus
  • 09:39 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 09:38 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:35 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:35 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:32 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:32 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:29 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus
  • 09:26 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:26 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus
  • 09:24 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 09:24 wm-bot: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus

2021-06-09

  • 17:33 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy (T281881)
  • 13:30 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 13:30 wm-bot: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:27 wm-bot: Rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:27 wm-bot: Finished rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:24 wm-bot: Rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:24 wm-bot: Finished rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:21 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:21 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 13:01 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus
  • 12:53 wm-bot: Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 12:53 wm-bot: Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus

2021-06-08

  • 23:19 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 (T281881)
  • 21:08 bstorm: downtiming grafana-labs for maintenance
  • 16:28 wm-bot: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 16:27 wm-bot: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 16:24 wm-bot: Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 16:24 wm-bot: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 16:22 wm-bot: Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 16:21 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 16:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus
  • 16:18 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 16:17 wm-bot: Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • 15:03 wm-bot: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:59 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:57 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:29 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:23 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus
  • 14:18 wm-bot: Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

2021-06-07

  • 14:27 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate T281399

2021-06-01

  • 13:12 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
  • 09:57 arturo: fix PTR record for 185.15.56.1 (T284025)
  • 09:56 arturo: fix PTR record for 185.15.56.1 (T248025)

2021-05-27

  • 14:58 wm-bot: Testing - cookbook ran by dcaro@vulcanus

2021-05-26

  • 19:10 andrewbogott: reimaging cloudvirt1018 to support local VM storage
  • 18:07 andrewbogott: draining cloudvirt1018, converting it to a local-storage host like cloudvirt1019 and 1020 -- T283296
  • 14:36 dcaro: Enabled syslog logging for osd.55 on eqiad ceph cluster for testing (T281247)
  • 14:36 dcaro: Enabled syslog logging on codfw ceph cluster (mon/osd/mgr) (T281247)
  • 11:26 arturo: [codfw1dev] purge old kernel packages in cloudvirt200[12]-dev
  • 11:03 arturo: created public flavor `g3.cores16.ram36.disk20` (even though it was requeste