You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27163 and previous config saved to /var/cache/conftool/dbconfig/20220429-231136-ladsgroup.json)
imported>Nhatminh01
(archive 52)
Line 1: Line 1:
== 2022-04-29 ==
* 23:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27163 and previous config saved to /var/cache/conftool/dbconfig/20220429-231136-ladsgroup.json
* 22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P27162 and previous config saved to /var/cache/conftool/dbconfig/20220429-225631-ladsgroup.json
* 22:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2003.codfw.wmnet with OS bullseye
* 22:48 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2004.codfw.wmnet with OS bullseye
* 22:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P27161 and previous config saved to /var/cache/conftool/dbconfig/20220429-224125-ladsgroup.json
* 22:37 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage
* 22:33 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage
* 22:28 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bullseye
* 22:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27160 and previous config saved to /var/cache/conftool/dbconfig/20220429-222620-ladsgroup.json
* 22:25 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2003.codfw.wmnet with OS bullseye
* 22:16 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bullseye
* 22:15 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2004.codfw.wmnet with OS bullseye
* 22:15 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bullseye
* 22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27159 and previous config saved to /var/cache/conftool/dbconfig/20220429-221331-ladsgroup.json
* 22:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 22:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 22:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27158 and previous config saved to /var/cache/conftool/dbconfig/20220429-221323-ladsgroup.json
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P27157 and previous config saved to /var/cache/conftool/dbconfig/20220429-215818-ladsgroup.json
* 21:54 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bullseye
* 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P27156 and previous config saved to /var/cache/conftool/dbconfig/20220429-214313-ladsgroup.json
* 21:42 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2002.codfw.wmnet with OS bullseye
* 21:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 21:29 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage
* 21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27155 and previous config saved to /var/cache/conftool/dbconfig/20220429-212808-ladsgroup.json
* 21:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 21:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 21:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 21:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 21:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27154 and previous config saved to /var/cache/conftool/dbconfig/20220429-212652-ladsgroup.json
* 21:26 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage
* 21:21 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2002.codfw.wmnet with OS bullseye
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27153 and previous config saved to /var/cache/conftool/dbconfig/20220429-211609-ladsgroup.json
* 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27152 and previous config saved to /var/cache/conftool/dbconfig/20220429-211601-ladsgroup.json
* 21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27151 and previous config saved to /var/cache/conftool/dbconfig/20220429-211146-ladsgroup.json
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P27150 and previous config saved to /var/cache/conftool/dbconfig/20220429-210055-ladsgroup.json
* 20:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27149 and previous config saved to /var/cache/conftool/dbconfig/20220429-205641-ladsgroup.json
* 20:46 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2002.codfw.wmnet with OS bullseye
* 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P27148 and previous config saved to /var/cache/conftool/dbconfig/20220429-204550-ladsgroup.json
* 20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27147 and previous config saved to /var/cache/conftool/dbconfig/20220429-204136-ladsgroup.json
* 20:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2001.codfw.wmnet with OS bullseye
* 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27146 and previous config saved to /var/cache/conftool/dbconfig/20220429-203045-ladsgroup.json
* 20:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27145 and previous config saved to /var/cache/conftool/dbconfig/20220429-202824-ladsgroup.json
* 20:19 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage
* 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27144 and previous config saved to /var/cache/conftool/dbconfig/20220429-201753-ladsgroup.json
* 20:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 20:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27143 and previous config saved to /var/cache/conftool/dbconfig/20220429-201745-ladsgroup.json
* 20:16 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage
* 20:15 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2002.codfw.wmnet with OS bullseye
* 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27142 and previous config saved to /var/cache/conftool/dbconfig/20220429-201319-ladsgroup.json
* 20:11 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2001.codfw.wmnet with OS bullseye
* 20:08 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2001.codfw.wmnet with OS bullseye
* 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P27141 and previous config saved to /var/cache/conftool/dbconfig/20220429-200240-ladsgroup.json
* 19:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27140 and previous config saved to /var/cache/conftool/dbconfig/20220429-195813-ladsgroup.json
* 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P27139 and previous config saved to /var/cache/conftool/dbconfig/20220429-194735-ladsgroup.json
* 19:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27138 and previous config saved to /var/cache/conftool/dbconfig/20220429-194308-ladsgroup.json
* 19:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27137 and previous config saved to /var/cache/conftool/dbconfig/20220429-194122-ladsgroup.json
* 19:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 19:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27136 and previous config saved to /var/cache/conftool/dbconfig/20220429-193649-ladsgroup.json
* 19:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 19:36 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host aqs2001.codfw.wmnet with OS bullseye
* 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27135 and previous config saved to /var/cache/conftool/dbconfig/20220429-193624-ladsgroup.json
* 19:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27134 and previous config saved to /var/cache/conftool/dbconfig/20220429-193549-ladsgroup.json
* 19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27133 and previous config saved to /var/cache/conftool/dbconfig/20220429-193230-ladsgroup.json
* 19:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P27132 and previous config saved to /var/cache/conftool/dbconfig/20220429-192119-ladsgroup.json
* 19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27131 and previous config saved to /var/cache/conftool/dbconfig/20220429-192044-ladsgroup.json
* 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27130 and previous config saved to /var/cache/conftool/dbconfig/20220429-191932-ladsgroup.json
* 19:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 19:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 19:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 19:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P27129 and previous config saved to /var/cache/conftool/dbconfig/20220429-190614-ladsgroup.json
* 19:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27128 and previous config saved to /var/cache/conftool/dbconfig/20220429-190539-ladsgroup.json
* 18:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27127 and previous config saved to /var/cache/conftool/dbconfig/20220429-185705-ladsgroup.json
* 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27126 and previous config saved to /var/cache/conftool/dbconfig/20220429-185109-ladsgroup.json
* 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27125 and previous config saved to /var/cache/conftool/dbconfig/20220429-185034-ladsgroup.json
* 18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27124 and previous config saved to /var/cache/conftool/dbconfig/20220429-184506-ladsgroup.json
* 18:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27123 and previous config saved to /var/cache/conftool/dbconfig/20220429-184411-ladsgroup.json
* 18:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27122 and previous config saved to /var/cache/conftool/dbconfig/20220429-184313-ladsgroup.json
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P27121 and previous config saved to /var/cache/conftool/dbconfig/20220429-184200-ladsgroup.json
* 18:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
* 18:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 18:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P27120 and previous config saved to /var/cache/conftool/dbconfig/20220429-183001-ladsgroup.json
* 18:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P27119 and previous config saved to /var/cache/conftool/dbconfig/20220429-182807-ladsgroup.json
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27118 and previous config saved to /var/cache/conftool/dbconfig/20220429-182714-ladsgroup.json
* 18:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 18:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27117 and previous config saved to /var/cache/conftool/dbconfig/20220429-182700-ladsgroup.json
* 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P27116 and previous config saved to /var/cache/conftool/dbconfig/20220429-182653-ladsgroup.json
* 18:21 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be1040.eqiad.wmnet
* 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P27115 and previous config saved to /var/cache/conftool/dbconfig/20220429-181456-ladsgroup.json
* 18:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P27114 and previous config saved to /var/cache/conftool/dbconfig/20220429-181302-ladsgroup.json
* 18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27113 and previous config saved to /var/cache/conftool/dbconfig/20220429-181153-ladsgroup.json
* 18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27112 and previous config saved to /var/cache/conftool/dbconfig/20220429-181145-ladsgroup.json
* 18:11 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be1040.eqiad.wmnet
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27111 and previous config saved to /var/cache/conftool/dbconfig/20220429-175951-ladsgroup.json
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27110 and previous config saved to /var/cache/conftool/dbconfig/20220429-175903-ladsgroup.json
* 17:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 17:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27109 and previous config saved to /var/cache/conftool/dbconfig/20220429-175855-ladsgroup.json
* 17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27108 and previous config saved to /var/cache/conftool/dbconfig/20220429-175841-ladsgroup.json
* 17:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27107 and previous config saved to /var/cache/conftool/dbconfig/20220429-175833-ladsgroup.json
* 17:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27106 and previous config saved to /var/cache/conftool/dbconfig/20220429-175757-ladsgroup.json
* 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27105 and previous config saved to /var/cache/conftool/dbconfig/20220429-175642-ladsgroup.json
* 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27104 and previous config saved to /var/cache/conftool/dbconfig/20220429-175122-ladsgroup.json
* 17:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27103 and previous config saved to /var/cache/conftool/dbconfig/20220429-175114-ladsgroup.json
* 17:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P27102 and previous config saved to /var/cache/conftool/dbconfig/20220429-174350-ladsgroup.json
* 17:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P27101 and previous config saved to /var/cache/conftool/dbconfig/20220429-174328-ladsgroup.json
* 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27100 and previous config saved to /var/cache/conftool/dbconfig/20220429-174136-ladsgroup.json
* 17:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T276292|T276292]])', diff saved to https://phabricator.wikimedia.org/P27099 and previous config saved to /var/cache/conftool/dbconfig/20220429-174129-ladsgroup.json
* 17:36 Amir1: killed bnwiki's refresh links recommendation ([[phab:T299021|T299021]])
* 17:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27098 and previous config saved to /var/cache/conftool/dbconfig/20220429-173609-ladsgroup.json
* 17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P27097 and previous config saved to /var/cache/conftool/dbconfig/20220429-172845-ladsgroup.json
* 17:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P27096 and previous config saved to /var/cache/conftool/dbconfig/20220429-172823-ladsgroup.json
* 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P27095 and previous config saved to /var/cache/conftool/dbconfig/20220429-172623-ladsgroup.json
* 17:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27094 and previous config saved to /var/cache/conftool/dbconfig/20220429-172104-ladsgroup.json
* 17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27093 and previous config saved to /var/cache/conftool/dbconfig/20220429-171658-ladsgroup.json
* 17:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 17:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 17:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27092 and previous config saved to /var/cache/conftool/dbconfig/20220429-171650-ladsgroup.json
* 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27091 and previous config saved to /var/cache/conftool/dbconfig/20220429-171339-ladsgroup.json
* 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27090 and previous config saved to /var/cache/conftool/dbconfig/20220429-171318-ladsgroup.json
* 17:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P27089 and previous config saved to /var/cache/conftool/dbconfig/20220429-171118-ladsgroup.json
* 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27088 and previous config saved to /var/cache/conftool/dbconfig/20220429-170559-ladsgroup.json
* 17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27087 and previous config saved to /var/cache/conftool/dbconfig/20220429-170205-ladsgroup.json
* 17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 17:02 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1002.eqiad.wmnet with OS buster
* 17:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P27086 and previous config saved to /var/cache/conftool/dbconfig/20220429-170145-ladsgroup.json
* 17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 17:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 17:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 17:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27085 and previous config saved to /var/cache/conftool/dbconfig/20220429-170125-ladsgroup.json
* 16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27084 and previous config saved to /var/cache/conftool/dbconfig/20220429-165939-ladsgroup.json
* 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 16:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27083 and previous config saved to /var/cache/conftool/dbconfig/20220429-165839-ladsgroup.json
* 16:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T276292|T276292]])', diff saved to https://phabricator.wikimedia.org/P27082 and previous config saved to /var/cache/conftool/dbconfig/20220429-165613-ladsgroup.json
* 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T276292|T276292]])', diff saved to https://phabricator.wikimedia.org/P27081 and previous config saved to /var/cache/conftool/dbconfig/20220429-165333-ladsgroup.json
* 16:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 16:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 16:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
* 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
* 16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27080 and previous config saved to /var/cache/conftool/dbconfig/20220429-165035-ladsgroup.json
* 16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P27079 and previous config saved to /var/cache/conftool/dbconfig/20220429-164640-ladsgroup.json
* 16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P27078 and previous config saved to /var/cache/conftool/dbconfig/20220429-164620-ladsgroup.json
* 16:43 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab1004.wikimedia.org with OS bullseye
* 16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27077 and previous config saved to /var/cache/conftool/dbconfig/20220429-164333-ladsgroup.json
* 16:41 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab1003.wikimedia.org with OS bullseye
* 16:37 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1004.eqiad.wmnet with OS bullseye
* 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P27076 and previous config saved to /var/cache/conftool/dbconfig/20220429-163530-ladsgroup.json
* 16:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1004.wikimedia.org with reason: host reimage
* 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27075 and previous config saved to /var/cache/conftool/dbconfig/20220429-163135-ladsgroup.json
* 16:31 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1003.eqiad.wmnet with OS bullseye
* 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P27074 and previous config saved to /var/cache/conftool/dbconfig/20220429-163115-ladsgroup.json
* 16:30 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1002.eqiad.wmnet with OS buster
* 16:29 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2042.codfw.wmnet with OS bullseye
* 16:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1003.wikimedia.org with reason: host reimage
* 16:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1004.wikimedia.org with reason: host reimage
* 16:29 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1002.eqiad.wmnet with OS bullseye
* 16:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner1002.eqiad.wmnet with OS bullseye
* 16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27073 and previous config saved to /var/cache/conftool/dbconfig/20220429-162828-ladsgroup.json
* 16:26 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1004.eqiad.wmnet with reason: host reimage
* 16:25 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1003.wikimedia.org with reason: host reimage
* 16:23 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1004.eqiad.wmnet with reason: host reimage
* 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P27072 and previous config saved to /var/cache/conftool/dbconfig/20220429-162025-ladsgroup.json
* 16:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1003.eqiad.wmnet with reason: host reimage
* 16:18 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab1004.wikimedia.org with OS bullseye
* 16:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage
* 16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27071 and previous config saved to /var/cache/conftool/dbconfig/20220429-161610-ladsgroup.json
* 16:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab1003.wikimedia.org with OS bullseye
* 16:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1003.eqiad.wmnet with reason: host reimage
* 16:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27070 and previous config saved to /var/cache/conftool/dbconfig/20220429-161400-ladsgroup.json
* 16:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 16:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27069 and previous config saved to /var/cache/conftool/dbconfig/20220429-161352-ladsgroup.json
* 16:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner1002.eqiad.wmnet with reason: host reimage
* 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27068 and previous config saved to /var/cache/conftool/dbconfig/20220429-161323-ladsgroup.json
* 16:12 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab-runner1004.eqiad.wmnet with OS bullseye
* 16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27067 and previous config saved to /var/cache/conftool/dbconfig/20220429-160702-ladsgroup.json
* 16:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 16:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 16:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 16:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27066 and previous config saved to /var/cache/conftool/dbconfig/20220429-160602-ladsgroup.json
* 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27065 and previous config saved to /var/cache/conftool/dbconfig/20220429-160520-ladsgroup.json
* 16:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab-runner1003.eqiad.wmnet with OS bullseye
* 16:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab-runner1002.eqiad.wmnet with OS bullseye
* 15:59 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P27064 and previous config saved to /var/cache/conftool/dbconfig/20220429-155846-ladsgroup.json
* 15:56 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:53 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27063 and previous config saved to /var/cache/conftool/dbconfig/20220429-155142-ladsgroup.json
* 15:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 15:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27062 and previous config saved to /var/cache/conftool/dbconfig/20220429-155134-ladsgroup.json
* 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27061 and previous config saved to /var/cache/conftool/dbconfig/20220429-155057-ladsgroup.json
* 15:45 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P27060 and previous config saved to /var/cache/conftool/dbconfig/20220429-154341-ladsgroup.json
* 15:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27059 and previous config saved to /var/cache/conftool/dbconfig/20220429-154253-ladsgroup.json
* 15:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 15:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 15:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27058 and previous config saved to /var/cache/conftool/dbconfig/20220429-154245-ladsgroup.json
* 15:41 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 15:37 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:36 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1002.eqiad.wmnet with OS bullseye
* 15:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P27057 and previous config saved to /var/cache/conftool/dbconfig/20220429-153629-ladsgroup.json
* 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27056 and previous config saved to /var/cache/conftool/dbconfig/20220429-153551-ladsgroup.json
* 15:34 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:31 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:29 jynus: update NIC firmware for backup1002 [[phab:T286722|T286722]] [[phab:T305446|T305446]]
* 15:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27055 and previous config saved to /var/cache/conftool/dbconfig/20220429-152836-ladsgroup.json
* 15:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P27054 and previous config saved to /var/cache/conftool/dbconfig/20220429-152740-ladsgroup.json
* 15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27053 and previous config saved to /var/cache/conftool/dbconfig/20220429-152628-ladsgroup.json
* 15:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 15:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27052 and previous config saved to /var/cache/conftool/dbconfig/20220429-152620-ladsgroup.json
* 15:25 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2042.codfw.wmnet with reason: host reimage
* 15:25 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:21 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2042.codfw.wmnet with reason: host reimage
* 15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P27051 and previous config saved to /var/cache/conftool/dbconfig/20220429-152124-ladsgroup.json
* 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27050 and previous config saved to /var/cache/conftool/dbconfig/20220429-152046-ladsgroup.json
* 15:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27049 and previous config saved to /var/cache/conftool/dbconfig/20220429-151424-ladsgroup.json
* 15:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 15:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 15:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 15:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 15:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27048 and previous config saved to /var/cache/conftool/dbconfig/20220429-151321-ladsgroup.json
* 15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P27047 and previous config saved to /var/cache/conftool/dbconfig/20220429-151235-ladsgroup.json
* 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27046 and previous config saved to /var/cache/conftool/dbconfig/20220429-151115-ladsgroup.json
* 15:07 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2042.codfw.wmnet with OS bullseye
* 15:07 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner1004.mgmt.eqiad.wmnet with reboot policy FORCED
* 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27045 and previous config saved to /var/cache/conftool/dbconfig/20220429-150619-ladsgroup.json
* 15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27044 and previous config saved to /var/cache/conftool/dbconfig/20220429-150417-ladsgroup.json
* 14:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27043 and previous config saved to /var/cache/conftool/dbconfig/20220429-145816-ladsgroup.json
* 14:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27042 and previous config saved to /var/cache/conftool/dbconfig/20220429-145730-ladsgroup.json
* 14:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27041 and previous config saved to /var/cache/conftool/dbconfig/20220429-145610-ladsgroup.json
* 14:52 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab-runner1004.mgmt.eqiad.wmnet with reboot policy FORCED
* 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27040 and previous config saved to /var/cache/conftool/dbconfig/20220429-145148-ladsgroup.json
* 14:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 14:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27039 and previous config saved to /var/cache/conftool/dbconfig/20220429-144947-ladsgroup.json
* 14:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 14:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P27038 and previous config saved to /var/cache/conftool/dbconfig/20220429-144912-ladsgroup.json
* 14:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 14:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 14:43 jayme@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-wikikube-rw,name=eqiad
* 14:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P27037 and previous config saved to /var/cache/conftool/dbconfig/20220429-144311-ladsgroup.json
* 14:43 jayme@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-wikikube-ro
* 14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27036 and previous config saved to /var/cache/conftool/dbconfig/20220429-144105-ladsgroup.json
* 14:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 14:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27035 and previous config saved to /var/cache/conftool/dbconfig/20220429-144028-ladsgroup.json
* 14:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27034 and previous config saved to /var/cache/conftool/dbconfig/20220429-143857-ladsgroup.json
* 14:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 14:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 14:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 14:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27033 and previous config saved to /var/cache/conftool/dbconfig/20220429-143844-ladsgroup.json
* 14:35 jayme@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P27032 and previous config saved to /var/cache/conftool/dbconfig/20220429-143407-ladsgroup.json
* 14:32 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2041.codfw.wmnet with OS bullseye
* 14:31 jayme@cumin1001: START - Cookbook sre.dns.netbox
* 14:29 jayme@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-wikikube-ro
* 14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27031 and previous config saved to /var/cache/conftool/dbconfig/20220429-142806-ladsgroup.json
* 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P27030 and previous config saved to /var/cache/conftool/dbconfig/20220429-142523-ladsgroup.json
* 14:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P27029 and previous config saved to /var/cache/conftool/dbconfig/20220429-142339-ladsgroup.json
* 14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27028 and previous config saved to /var/cache/conftool/dbconfig/20220429-142142-ladsgroup.json
* 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:19 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2041.codfw.wmnet with reason: host reimage
* 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27027 and previous config saved to /var/cache/conftool/dbconfig/20220429-141902-ladsgroup.json
* 14:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27026 and previous config saved to /var/cache/conftool/dbconfig/20220429-141633-ladsgroup.json
* 14:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 14:16 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2041.codfw.wmnet with reason: host reimage
* 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P27025 and previous config saved to /var/cache/conftool/dbconfig/20220429-141017-ladsgroup.json
* 14:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P27024 and previous config saved to /var/cache/conftool/dbconfig/20220429-140834-ladsgroup.json
* 13:56 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2041.codfw.wmnet with OS bullseye
* 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27023 and previous config saved to /var/cache/conftool/dbconfig/20220429-135511-ladsgroup.json
* 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27022 and previous config saved to /var/cache/conftool/dbconfig/20220429-135329-ladsgroup.json
* 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27021 and previous config saved to /var/cache/conftool/dbconfig/20220429-135121-ladsgroup.json
* 13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27020 and previous config saved to /var/cache/conftool/dbconfig/20220429-135111-ladsgroup.json
* 13:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:42 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
* 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P27019 and previous config saved to /var/cache/conftool/dbconfig/20220429-133606-ladsgroup.json
* 13:35 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
* 13:32 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2040.codfw.wmnet
* 13:26 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be2040.codfw.wmnet
* 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27018 and previous config saved to /var/cache/conftool/dbconfig/20220429-132619-ladsgroup.json
* 13:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 13:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 13:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27017 and previous config saved to /var/cache/conftool/dbconfig/20220429-132611-ladsgroup.json
* 13:25 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2040.codfw.wmnet
* 13:21 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-be2040.codfw.wmnet
* 13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P27016 and previous config saved to /var/cache/conftool/dbconfig/20220429-132101-ladsgroup.json
* 13:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P27015 and previous config saved to /var/cache/conftool/dbconfig/20220429-131106-ladsgroup.json
* 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27014 and previous config saved to /var/cache/conftool/dbconfig/20220429-130556-ladsgroup.json
* 12:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P27013 and previous config saved to /var/cache/conftool/dbconfig/20220429-125601-ladsgroup.json
* 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4004.ulsfo.wmnet
* 12:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T298295|T298295]])', diff saved to https://phabricator.wikimedia.org/P27012 and previous config saved to /var/cache/conftool/dbconfig/20220429-125146-ladsgroup.json
* 12:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 12:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 12:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti4004.ulsfo.wmnet
* 12:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2003.codfw.wmnet with OS bullseye
* 12:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27011 and previous config saved to /var/cache/conftool/dbconfig/20220429-124056-ladsgroup.json
* 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev2003.codfw.wmnet
* 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev2003.codfw.wmnet
* 12:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2003.codfw.wmnet with reason: host reimage
* 12:30 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2005-dev.codfw.wmnet with OS bullseye
* 12:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2003.codfw.wmnet with reason: host reimage
* 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P27010 and previous config saved to /var/cache/conftool/dbconfig/20220429-122805-ladsgroup.json
* 12:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 12:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev2002.codfw.wmnet
* 12:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev2002.codfw.wmnet
* 12:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev2001.codfw.wmnet
* 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2003.codfw.wmnet with OS bullseye
* 12:12 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
* 12:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev2001.codfw.wmnet
* 12:09 dcaro@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
* 12:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner1003.mgmt.eqiad.wmnet with reboot policy FORCED
* 12:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner1002.mgmt.eqiad.wmnet with reboot policy FORCED
* 12:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab1004.mgmt.eqiad.wmnet with reboot policy FORCED
* 11:57 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab1003.mgmt.eqiad.wmnet with reboot policy FORCED
* 11:51 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab-runner1003.mgmt.eqiad.wmnet with reboot policy FORCED
* 11:51 dcaro@cumin1001: START - Cookbook sre.hosts.reimage for host cloudnet2005-dev.codfw.wmnet with OS bullseye
* 11:50 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab-runner1002.mgmt.eqiad.wmnet with reboot policy FORCED
* 11:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab1004.mgmt.eqiad.wmnet with reboot policy FORCED
* 11:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host gitlab1003.mgmt.eqiad.wmnet with reboot policy FORCED
* 10:52 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 10:30 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1040.eqiad.wmnet with OS bullseye
* 10:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1040.eqiad.wmnet with reason: host reimage
* 10:08 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1040.eqiad.wmnet with reason: host reimage
* 10:06 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1164.eqiad.wmnet with OS bullseye
* 09:57 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be1040.eqiad.wmnet with OS bullseye
* 09:57 moritzm: drain ganeti-test2003 [[phab:T306499|T306499]]
* 09:56 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2006-dev.codfw.wmnet with OS bullseye
* 09:42 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS bullseye
* 09:42 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage
* 09:39 dcaro@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage
* 09:36 kormat@cumin1001: dbctl commit (dc=all): 'db1164 depooling: Rebooting for [[phab:T303171|T303171]]', diff saved to https://phabricator.wikimedia.org/P27008 and previous config saved to /var/cache/conftool/dbconfig/20220429-093613-kormat.json
* 09:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1164.eqiad.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 09:36 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1164.eqiad.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 09:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2002.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 09:33 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2002.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 09:27 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2040.codfw.wmnet with OS bullseye
* 09:25 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 09:20 dcaro@cumin1001: START - Cookbook sre.hosts.reimage for host cloudnet2006-dev.codfw.wmnet with OS bullseye
* 09:14 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: Reboot [[phab:T303171|T303171]]', diff saved to https://phabricator.wikimedia.org/P27007 and previous config saved to /var/cache/conftool/dbconfig/20220429-091401-kormat.json
* 09:02 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2040.codfw.wmnet with reason: host reimage
* 08:58 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: Reboot [[phab:T303171|T303171]]', diff saved to https://phabricator.wikimedia.org/P27006 and previous config saved to /var/cache/conftool/dbconfig/20220429-085858-kormat.json
* 08:58 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2040.codfw.wmnet with reason: host reimage
* 08:46 dcausse: restarting blazegraph on wdqs1006 (deadlocked for 18hours, [[phab:T242453|T242453]])
* 08:43 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 50%: Reboot [[phab:T303171|T303171]]', diff saved to https://phabricator.wikimedia.org/P27005 and previous config saved to /var/cache/conftool/dbconfig/20220429-084354-kormat.json
* 08:33 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw1323.eqiad.wmnet
* 08:28 kormat@cumin1001: dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: Reboot [[phab:T303171|T303171]]', diff saved to https://phabricator.wikimedia.org/P27004 and previous config saved to /var/cache/conftool/dbconfig/20220429-082850-kormat.json
* 08:27 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1163.eqiad.wmnet with OS bullseye
* 08:27 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 08:21 jelto: scap pull on mw1323
* 08:16 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1163.eqiad.wmnet with reason: host reimage
* 08:14 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
* 08:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1163.eqiad.wmnet with reason: host reimage
* 08:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
* 08:03 jelto@cumin1001: conftool action : set/pooled=no; selector: name=mw1323.eqiad.wmnet
* 08:02 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db1163.eqiad.wmnet with OS bullseye
* 08:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
* 08:00 kormat@cumin1001: dbctl commit (dc=all): 'db1163 depooling: Rebooting for [[phab:T303171|T303171]]', diff saved to https://phabricator.wikimedia.org/P27003 and previous config saved to /var/cache/conftool/dbconfig/20220429-080038-kormat.json
* 08:00 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1163.eqiad.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 08:00 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1163.eqiad.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 07:55 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2103.codfw.wmnet with OS bullseye
* 07:51 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 07:50 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
* 07:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2103.codfw.wmnet with reason: host reimage
* 07:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2103.codfw.wmnet with reason: host reimage
* 07:27 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2103.codfw.wmnet with OS bullseye
* 07:26 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 07:09 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2103.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 07:09 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2103.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 07:08 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: Reimaging db2103 [[phab:T303171|T303171]]
* 07:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 14 hosts with reason: Reimaging db2103 [[phab:T303171|T303171]]
* 06:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1132 [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P27001 and previous config saved to /var/cache/conftool/dbconfig/20220429-063019-marostegui.json
* 03:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P27000 and previous config saved to /var/cache/conftool/dbconfig/20220429-035843-ladsgroup.json
* 03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26999 and previous config saved to /var/cache/conftool/dbconfig/20220429-035415-ladsgroup.json
* 03:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26998 and previous config saved to /var/cache/conftool/dbconfig/20220429-034338-ladsgroup.json
* 03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26997 and previous config saved to /var/cache/conftool/dbconfig/20220429-033910-ladsgroup.json
* 03:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26996 and previous config saved to /var/cache/conftool/dbconfig/20220429-032833-ladsgroup.json
* 03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26995 and previous config saved to /var/cache/conftool/dbconfig/20220429-032405-ladsgroup.json
* 03:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26994 and previous config saved to /var/cache/conftool/dbconfig/20220429-031328-ladsgroup.json
* 03:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26993 and previous config saved to /var/cache/conftool/dbconfig/20220429-030900-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26992 and previous config saved to /var/cache/conftool/dbconfig/20220429-030447-ladsgroup.json
* 03:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 03:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26991 and previous config saved to /var/cache/conftool/dbconfig/20220429-030439-ladsgroup.json
* 03:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26990 and previous config saved to /var/cache/conftool/dbconfig/20220429-030303-ladsgroup.json
* 03:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 03:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26989 and previous config saved to /var/cache/conftool/dbconfig/20220429-030250-ladsgroup.json
* 02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26988 and previous config saved to /var/cache/conftool/dbconfig/20220429-024934-ladsgroup.json
* 02:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26987 and previous config saved to /var/cache/conftool/dbconfig/20220429-024745-ladsgroup.json
* 02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26986 and previous config saved to /var/cache/conftool/dbconfig/20220429-023429-ladsgroup.json
* 02:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2012.mgmt.codfw.wmnet with reboot policy FORCED
* 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26985 and previous config saved to /var/cache/conftool/dbconfig/20220429-023240-ladsgroup.json
* 02:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2011.mgmt.codfw.wmnet with reboot policy FORCED
* 02:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26984 and previous config saved to /var/cache/conftool/dbconfig/20220429-021924-ladsgroup.json
* 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26983 and previous config saved to /var/cache/conftool/dbconfig/20220429-021735-ladsgroup.json
* 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26982 and previous config saved to /var/cache/conftool/dbconfig/20220429-021710-ladsgroup.json
* 02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 02:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 02:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 02:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26981 and previous config saved to /var/cache/conftool/dbconfig/20220429-021657-ladsgroup.json
* 02:09 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2012.mgmt.codfw.wmnet with reboot policy FORCED
* 02:08 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2011.mgmt.codfw.wmnet with reboot policy FORCED
* 02:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2010.mgmt.codfw.wmnet with reboot policy FORCED
* 02:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2009.mgmt.codfw.wmnet with reboot policy FORCED
* 02:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26980 and previous config saved to /var/cache/conftool/dbconfig/20220429-020705-ladsgroup.json
* 02:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 02:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 02:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26979 and previous config saved to /var/cache/conftool/dbconfig/20220429-020652-ladsgroup.json
* 02:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26978 and previous config saved to /var/cache/conftool/dbconfig/20220429-020151-ladsgroup.json
* 01:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26977 and previous config saved to /var/cache/conftool/dbconfig/20220429-015147-ladsgroup.json
* 01:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26976 and previous config saved to /var/cache/conftool/dbconfig/20220429-014646-ladsgroup.json
* 01:40 ejegg: updated fundraising IPN listener standalone (SmashPig) from {{Gerrit|a5d785fd}} to {{Gerrit|ffe5066d}}
* 01:37 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2010.mgmt.codfw.wmnet with reboot policy FORCED
* 01:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26975 and previous config saved to /var/cache/conftool/dbconfig/20220429-013642-ladsgroup.json
* 01:36 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2009.mgmt.codfw.wmnet with reboot policy FORCED
* 01:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2008.mgmt.codfw.wmnet with reboot policy FORCED
* 01:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26974 and previous config saved to /var/cache/conftool/dbconfig/20220429-013141-ladsgroup.json
* 01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26973 and previous config saved to /var/cache/conftool/dbconfig/20220429-012827-ladsgroup.json
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 01:28 ejegg: updated Fundraising CiviCRM from {{Gerrit|a841cf55}} to {{Gerrit|852c1969}}
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 01:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26972 and previous config saved to /var/cache/conftool/dbconfig/20220429-012713-ladsgroup.json
* 01:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26971 and previous config saved to /var/cache/conftool/dbconfig/20220429-012137-ladsgroup.json
* 01:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26970 and previous config saved to /var/cache/conftool/dbconfig/20220429-011207-ladsgroup.json
* 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26969 and previous config saved to /var/cache/conftool/dbconfig/20220429-011046-ladsgroup.json
* 01:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 01:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26968 and previous config saved to /var/cache/conftool/dbconfig/20220429-011033-ladsgroup.json
* 01:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2007.mgmt.codfw.wmnet with reboot policy FORCED
* 00:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26967 and previous config saved to /var/cache/conftool/dbconfig/20220429-005702-ladsgroup.json
* 00:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2008.mgmt.codfw.wmnet with reboot policy FORCED
* 00:56 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 00:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26966 and previous config saved to /var/cache/conftool/dbconfig/20220429-005528-ladsgroup.json
* 00:53 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 00:47 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2007.mgmt.codfw.wmnet with reboot policy FORCED
* 00:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 00:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 00:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26965 and previous config saved to /var/cache/conftool/dbconfig/20220429-004157-ladsgroup.json
* 00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26964 and previous config saved to /var/cache/conftool/dbconfig/20220429-004023-ladsgroup.json
* 00:38 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 00:37 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 00:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26963 and previous config saved to /var/cache/conftool/dbconfig/20220429-002518-ladsgroup.json
* 00:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26962 and previous config saved to /var/cache/conftool/dbconfig/20220429-001840-ladsgroup.json
* 00:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 00:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 00:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26961 and previous config saved to /var/cache/conftool/dbconfig/20220429-001832-ladsgroup.json
* 00:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26960 and previous config saved to /var/cache/conftool/dbconfig/20220429-001333-ladsgroup.json
* 00:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26959 and previous config saved to /var/cache/conftool/dbconfig/20220429-001320-ladsgroup.json
* 00:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26958 and previous config saved to /var/cache/conftool/dbconfig/20220429-000327-ladsgroup.json


== 2022-04-28 ==
* 23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26957 and previous config saved to /var/cache/conftool/dbconfig/20220428-235815-ladsgroup.json
* 23:49 dzahn@deploy1002: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 23:49 dzahn@deploy1002: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 23:48 dzahn@deploy1002: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26956 and previous config saved to /var/cache/conftool/dbconfig/20220428-234822-ladsgroup.json
* 23:47 dzahn@deploy1002: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 23:44 dzahn@deploy1002: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26955 and previous config saved to /var/cache/conftool/dbconfig/20220428-234310-ladsgroup.json
* 23:42 dzahn@deploy1002: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 23:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26954 and previous config saved to /var/cache/conftool/dbconfig/20220428-233317-ladsgroup.json
* 23:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26953 and previous config saved to /var/cache/conftool/dbconfig/20220428-233103-ladsgroup.json
* 23:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 23:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 23:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26952 and previous config saved to /var/cache/conftool/dbconfig/20220428-233055-ladsgroup.json
* 23:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26951 and previous config saved to /var/cache/conftool/dbconfig/20220428-232805-ladsgroup.json
* 23:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 23:18 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26950 and previous config saved to /var/cache/conftool/dbconfig/20220428-231714-ladsgroup.json
* 23:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 23:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 23:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2003.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2004.mgmt.codfw.wmnet with reboot policy FORCED
* 23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26949 and previous config saved to /var/cache/conftool/dbconfig/20220428-231701-ladsgroup.json
* 23:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26948 and previous config saved to /var/cache/conftool/dbconfig/20220428-231550-ladsgroup.json
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26947 and previous config saved to /var/cache/conftool/dbconfig/20220428-230156-ladsgroup.json
* 23:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26946 and previous config saved to /var/cache/conftool/dbconfig/20220428-230045-ladsgroup.json
* 22:54 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2004.mgmt.codfw.wmnet with reboot policy FORCED
* 22:54 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2003.mgmt.codfw.wmnet with reboot policy FORCED
* 22:48 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2002.mgmt.codfw.wmnet with reboot policy FORCED
* 22:48 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2001.mgmt.codfw.wmnet with reboot policy FORCED
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26945 and previous config saved to /var/cache/conftool/dbconfig/20220428-224650-ladsgroup.json
* 22:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26944 and previous config saved to /var/cache/conftool/dbconfig/20220428-224540-ladsgroup.json
* 22:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26943 and previous config saved to /var/cache/conftool/dbconfig/20220428-224426-ladsgroup.json
* 22:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 22:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 22:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26942 and previous config saved to /var/cache/conftool/dbconfig/20220428-224417-ladsgroup.json
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26941 and previous config saved to /var/cache/conftool/dbconfig/20220428-223145-ladsgroup.json
* 22:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26940 and previous config saved to /var/cache/conftool/dbconfig/20220428-222912-ladsgroup.json
* 22:21 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2002.mgmt.codfw.wmnet with reboot policy FORCED
* 22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26939 and previous config saved to /var/cache/conftool/dbconfig/20220428-222035-ladsgroup.json
* 22:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 22:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26938 and previous config saved to /var/cache/conftool/dbconfig/20220428-222022-ladsgroup.json
* 22:20 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2001.mgmt.codfw.wmnet with reboot policy FORCED
* 22:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26937 and previous config saved to /var/cache/conftool/dbconfig/20220428-221407-ladsgroup.json
* 22:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26936 and previous config saved to /var/cache/conftool/dbconfig/20220428-220517-ladsgroup.json
* 22:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26935 and previous config saved to /var/cache/conftool/dbconfig/20220428-220450-ladsgroup.json
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26934 and previous config saved to /var/cache/conftool/dbconfig/20220428-215902-ladsgroup.json
* 21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26933 and previous config saved to /var/cache/conftool/dbconfig/20220428-215547-ladsgroup.json
* 21:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 21:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 21:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26932 and previous config saved to /var/cache/conftool/dbconfig/20220428-215012-ladsgroup.json
* 21:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26931 and previous config saved to /var/cache/conftool/dbconfig/20220428-214945-ladsgroup.json
* 21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26930 and previous config saved to /var/cache/conftool/dbconfig/20220428-213507-ladsgroup.json
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26929 and previous config saved to /var/cache/conftool/dbconfig/20220428-213440-ladsgroup.json
* 21:27 tgr: UTC late deploys done
* 21:26 tgr@deploy1002: Synchronized php-1.39.0-wmf.9/extensions/GrowthExperiments/includes/VariantHooks.php: Backport: [[gerrit:787461{{!}}Video landing page: Record campaign parameter for control users (T303785)]] (duration: 00m 54s)
* 21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26928 and previous config saved to /var/cache/conftool/dbconfig/20220428-212331-ladsgroup.json
* 21:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 21:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 21:20 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26927 and previous config saved to /var/cache/conftool/dbconfig/20220428-211934-ladsgroup.json
* 21:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26926 and previous config saved to /var/cache/conftool/dbconfig/20220428-211727-ladsgroup.json
* 21:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 21:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 21:15 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 21:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26925 and previous config saved to /var/cache/conftool/dbconfig/20220428-211507-ladsgroup.json
* 21:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 21:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 21:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 21:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26924 and previous config saved to /var/cache/conftool/dbconfig/20220428-210002-ladsgroup.json
* 20:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26923 and previous config saved to /var/cache/conftool/dbconfig/20220428-204457-ladsgroup.json
* 20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26922 and previous config saved to /var/cache/conftool/dbconfig/20220428-202952-ladsgroup.json
* 20:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:09 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:07 brennen: no trainee attendees for backport & config session; tgr self-serving some patches, calling end of training window
* 20:06 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26921 and previous config saved to /var/cache/conftool/dbconfig/20220428-193109-ladsgroup.json
* 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26920 and previous config saved to /var/cache/conftool/dbconfig/20220428-192938-ladsgroup.json
* 19:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 19:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26919 and previous config saved to /var/cache/conftool/dbconfig/20220428-192930-ladsgroup.json
* 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26918 and previous config saved to /var/cache/conftool/dbconfig/20220428-191604-ladsgroup.json
* 19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26917 and previous config saved to /var/cache/conftool/dbconfig/20220428-191425-ladsgroup.json
* 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26915 and previous config saved to /var/cache/conftool/dbconfig/20220428-190059-ladsgroup.json
* 18:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26914 and previous config saved to /var/cache/conftool/dbconfig/20220428-185920-ladsgroup.json
* 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26913 and previous config saved to /var/cache/conftool/dbconfig/20220428-184554-ladsgroup.json
* 18:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26912 and previous config saved to /var/cache/conftool/dbconfig/20220428-184415-ladsgroup.json
* 18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26911 and previous config saved to /var/cache/conftool/dbconfig/20220428-184338-ladsgroup.json
* 18:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 18:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26910 and previous config saved to /var/cache/conftool/dbconfig/20220428-184330-ladsgroup.json
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26909 and previous config saved to /var/cache/conftool/dbconfig/20220428-184207-ladsgroup.json
* 18:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26908 and previous config saved to /var/cache/conftool/dbconfig/20220428-184159-ladsgroup.json
* 18:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26907 and previous config saved to /var/cache/conftool/dbconfig/20220428-182825-ladsgroup.json
* 18:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1024.eqiad.wmnet with OS buster
* 18:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26906 and previous config saved to /var/cache/conftool/dbconfig/20220428-182654-ladsgroup.json
* 18:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1023.eqiad.wmnet with OS buster
* 18:21 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1022.eqiad.wmnet with OS buster
* 18:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1021.eqiad.wmnet with OS buster
* 18:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1020.eqiad.wmnet with OS buster
* 18:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:16 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1024.eqiad.wmnet with reason: host reimage
* 18:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1023.eqiad.wmnet with reason: host reimage
* 18:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26905 and previous config saved to /var/cache/conftool/dbconfig/20220428-181320-ladsgroup.json
* 18:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1019.eqiad.wmnet with OS buster
* 18:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26904 and previous config saved to /var/cache/conftool/dbconfig/20220428-181149-ladsgroup.json
* 18:11 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1018.eqiad.wmnet with OS buster
* 18:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1024.eqiad.wmnet with reason: host reimage
* 18:10 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1023.eqiad.wmnet with reason: host reimage
* 18:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1017.eqiad.wmnet with OS buster
* 18:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1022.eqiad.wmnet with reason: host reimage
* 18:07 brennen@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]]
* 18:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2002.wikimedia.org with OS bullseye
* 18:05 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1021.eqiad.wmnet with reason: host reimage
* 18:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2003.wikimedia.org with OS bullseye
* 18:03 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1020.eqiad.wmnet with reason: host reimage
* 18:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1022.eqiad.wmnet with reason: host reimage
* 18:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1014.eqiad.wmnet with OS buster
* 18:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1011.eqiad.wmnet with OS buster
* 18:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1012.eqiad.wmnet with OS buster
* 18:01 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers, logs fairly clear, proceeding to all wikis as soon as i finish this burrito
* 18:00 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1019.eqiad.wmnet with reason: host reimage
* 18:00 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1021.eqiad.wmnet with reason: host reimage
* 17:59 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1024.eqiad.wmnet with OS buster
* 17:59 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1018.eqiad.wmnet with reason: host reimage
* 17:59 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1023.eqiad.wmnet with OS buster
* 17:59 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host parse1016.eqiad.wmnet with OS buster
* 17:58 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1015.eqiad.wmnet with OS buster
* 17:58 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1020.eqiad.wmnet with reason: host reimage
* 17:58 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1032.eqiad.wmnet with OS buster
* 17:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26903 and previous config saved to /var/cache/conftool/dbconfig/20220428-175815-ladsgroup.json
* 17:57 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1017.eqiad.wmnet with reason: host reimage
* 17:57 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1018.eqiad.wmnet with reason: host reimage
* 17:56 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1019.eqiad.wmnet with reason: host reimage
* 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26902 and previous config saved to /var/cache/conftool/dbconfig/20220428-175644-ladsgroup.json
* 17:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26901 and previous config saved to /var/cache/conftool/dbconfig/20220428-175559-ladsgroup.json
* 17:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 17:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26900 and previous config saved to /var/cache/conftool/dbconfig/20220428-175551-ladsgroup.json
* 17:55 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1013.eqiad.wmnet with OS buster
* 17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26899 and previous config saved to /var/cache/conftool/dbconfig/20220428-175436-ladsgroup.json
* 17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26898 and previous config saved to /var/cache/conftool/dbconfig/20220428-175423-ladsgroup.json
* 17:53 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage
* 17:53 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1017.eqiad.wmnet with reason: host reimage
* 17:51 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1022.eqiad.wmnet with OS buster
* 17:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage
* 17:51 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1031.eqiad.wmnet with OS buster
* 17:51 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1014.eqiad.wmnet with reason: host reimage
* 17:49 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1021.eqiad.wmnet with OS buster
* 17:48 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1030.eqiad.wmnet with OS buster
* 17:48 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1012.eqiad.wmnet with reason: host reimage
* 17:48 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1011.eqiad.wmnet with reason: host reimage
* 17:47 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1020.eqiad.wmnet with OS buster
* 17:47 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1010.eqiad.wmnet with OS buster
* 17:46 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage
* 17:46 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage
* 17:46 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1016.eqiad.wmnet with reason: host reimage
* 17:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1019.eqiad.wmnet with OS buster
* 17:45 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti1032.eqiad.wmnet with reason: host reimage
* 17:45 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1015.eqiad.wmnet with reason: host reimage
* 17:45 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1009.eqiad.wmnet with OS buster
* 17:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1018.eqiad.wmnet with OS buster
* 17:44 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1029.eqiad.wmnet with OS buster
* 17:43 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1013.eqiad.wmnet with reason: host reimage
* 17:42 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1017.eqiad.wmnet with OS buster
* 17:41 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1016.eqiad.wmnet with reason: host reimage
* 17:41 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1015.eqiad.wmnet with reason: host reimage
* 17:41 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1014.eqiad.wmnet with reason: host reimage
* 17:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26897 and previous config saved to /var/cache/conftool/dbconfig/20220428-174046-ladsgroup.json
* 17:40 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parse1016.eqiad.wmnet with OS buster
* 17:40 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
* 17:39 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti1031.eqiad.wmnet with reason: host reimage
* 17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26896 and previous config saved to /var/cache/conftool/dbconfig/20220428-173918-ladsgroup.json
* 17:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1013.eqiad.wmnet with reason: host reimage
* 17:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1012.eqiad.wmnet with reason: host reimage
* 17:38 dancy: testing logging: [[gerrit:9999]]
* 17:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1011.eqiad.wmnet with reason: host reimage
* 17:37 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1030.eqiad.wmnet with reason: host reimage
* 17:36 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parse1016.eqiad.wmnet with OS buster
* 17:36 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
* 17:36 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host parse1016.eqiad.wmnet with OS buster
* 17:35 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
* 17:35 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1032.eqiad.wmnet with reason: host reimage
* 17:35 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1010.eqiad.wmnet with reason: host reimage
* 17:32 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1029.eqiad.wmnet with reason: host reimage
* 17:31 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
* 17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1016.eqiad.wmnet with OS buster
* 17:30 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1007.eqiad.wmnet with OS buster
* 17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1010.eqiad.wmnet with reason: host reimage
* 17:30 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab2002.wikimedia.org with OS bullseye
* 17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1015.eqiad.wmnet with OS buster
* 17:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1014.eqiad.wmnet with OS buster
* 17:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1003.eqiad.wmnet with OS buster
* 17:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1008.eqiad.wmnet with OS buster
* 17:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1031.eqiad.wmnet with reason: host reimage
* 17:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1030.eqiad.wmnet with reason: host reimage
* 17:29 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1029.eqiad.wmnet with reason: host reimage
* 17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1013.eqiad.wmnet with OS buster
* 17:27 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1009.eqiad.wmnet with reason: host reimage
* 17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1012.eqiad.wmnet with OS buster
* 17:27 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1011.eqiad.wmnet with OS buster
* 17:26 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1005.eqiad.wmnet with OS buster
* 17:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1004.eqiad.wmnet with OS buster
* 17:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26895 and previous config saved to /var/cache/conftool/dbconfig/20220428-172540-ladsgroup.json
* 17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26894 and previous config saved to /var/cache/conftool/dbconfig/20220428-172413-ladsgroup.json
* 17:24 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1009.eqiad.wmnet with reason: host reimage
* 17:23 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1006.eqiad.wmnet with OS buster
* 17:19 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1010.eqiad.wmnet with OS buster
* 17:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1007.eqiad.wmnet with reason: host reimage
* 17:19 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1002.eqiad.wmnet with OS buster
* 17:18 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1032.eqiad.wmnet with OS buster
* 17:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1031.eqiad.wmnet with OS buster
* 17:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1030.eqiad.wmnet with OS buster
* 17:17 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti1029.eqiad.wmnet with OS buster
* 17:17 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1003.eqiad.wmnet with reason: host reimage
* 17:16 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye
* 17:16 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1008.eqiad.wmnet with reason: host reimage
* 17:14 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1008.eqiad.wmnet with reason: host reimage
* 17:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1007.eqiad.wmnet with reason: host reimage
* 17:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1005.eqiad.wmnet with reason: host reimage
* 17:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1003.eqiad.wmnet with reason: host reimage
* 17:13 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1009.eqiad.wmnet with OS buster
* 17:12 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host parse1001.eqiad.wmnet with OS buster
* 17:11 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1004.eqiad.wmnet with reason: host reimage
* 17:11 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on parse1006.eqiad.wmnet with reason: host reimage
* 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26893 and previous config saved to /var/cache/conftool/dbconfig/20220428-171035-ladsgroup.json
* 17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26892 and previous config saved to /var/cache/conftool/dbconfig/20220428-170908-ladsgroup.json
* 17:08 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1002.eqiad.wmnet with reason: host reimage
* 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26891 and previous config saved to /var/cache/conftool/dbconfig/20220428-170820-ladsgroup.json
* 17:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 17:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 17:07 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1006.eqiad.wmnet with reason: host reimage
* 17:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26890 and previous config saved to /var/cache/conftool/dbconfig/20220428-170700-ladsgroup.json
* 17:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 17:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26889 and previous config saved to /var/cache/conftool/dbconfig/20220428-170652-ladsgroup.json
* 17:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:06 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1005.eqiad.wmnet with reason: host reimage
* 17:06 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1004.eqiad.wmnet with reason: host reimage
* 17:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:05 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1002.eqiad.wmnet with reason: host reimage
* 17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1008.eqiad.wmnet with OS buster
* 17:03 dancy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:03 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1007.eqiad.wmnet with OS buster
* 17:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1003.eqiad.wmnet with OS buster
* 17:02 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
* 17:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse1001.eqiad.wmnet with reason: host reimage
* 16:57 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on parse1001.eqiad.wmnet with reason: host reimage
* 16:57 dancy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:56 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1006.eqiad.wmnet with OS buster
* 16:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1005.eqiad.wmnet with OS buster
* 16:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1004.eqiad.wmnet with OS buster
* 16:54 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1002.eqiad.wmnet with OS buster
* 16:53 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2004.codfw.wmnet with OS bullseye
* 16:53 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1032.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:52 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1030.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:52 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1029.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26887 and previous config saved to /var/cache/conftool/dbconfig/20220428-165148-ladsgroup.json
* 16:46 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host parse1001.eqiad.wmnet with OS buster
* 16:43 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:42 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2004.codfw.wmnet with reason: host reimage
* 16:41 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1031.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:40 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1032.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:39 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1031.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:39 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2004.codfw.wmnet with reason: host reimage
* 16:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1030.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:38 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host ganeti1029.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:37 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26886 and previous config saved to /var/cache/conftool/dbconfig/20220428-163643-ladsgroup.json
* 16:33 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2003.codfw.wmnet with OS bullseye
* 16:31 dancy@deploy1002: Started scap: testing mediawiki container image build and deploy
* 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26885 and previous config saved to /var/cache/conftool/dbconfig/20220428-162159-ladsgroup.json
* 16:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26884 and previous config saved to /var/cache/conftool/dbconfig/20220428-162138-ladsgroup.json
* 16:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2003.codfw.wmnet with reason: host reimage
* 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26883 and previous config saved to /var/cache/conftool/dbconfig/20220428-162047-ladsgroup.json
* 16:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 16:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26882 and previous config saved to /var/cache/conftool/dbconfig/20220428-162039-ladsgroup.json
* 16:20 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2004.codfw.wmnet with OS bullseye
* 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26881 and previous config saved to /var/cache/conftool/dbconfig/20220428-161828-ladsgroup.json
* 16:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab-runner2002.codfw.wmnet with OS bullseye
* 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 16:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 16:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26880 and previous config saved to /var/cache/conftool/dbconfig/20220428-161748-ladsgroup.json
* 16:17 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2003.codfw.wmnet with reason: host reimage
* 16:10 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:07 dcaro@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudbackup2002.codfw.wmnet
* 16:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab-runner2002.codfw.wmnet with reason: host reimage
* 16:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26879 and previous config saved to /var/cache/conftool/dbconfig/20220428-160533-ladsgroup.json
* 16:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26878 and previous config saved to /var/cache/conftool/dbconfig/20220428-160243-ladsgroup.json
* 16:02 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab-runner2002.codfw.wmnet with reason: host reimage
* 16:01 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1015.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:01 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host parse1003.mgmt.eqiad.wmnet with reboot policy FORCED
* 15:59 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2003.codfw.wmnet with OS bullseye
* 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26877 and previous config saved to /var/cache/conftool/dbconfig/20220428-155028-ladsgroup.json
* 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26876 and previous config saved to /var/cache/conftool/dbconfig/20220428-154738-ladsgroup.json
* 15:45 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host parse1003.mgmt.eqiad.wmnet with reboot policy FORCED
* 15:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2002.codfw.wmnet with OS bullseye
* 15:39 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2072.codfw.wmnet with OS bullseye
* 15:38 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2130.codfw.wmnet with OS bullseye
* 15:38 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2112.codfw.wmnet with OS bullseye
* 15:35 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2145.codfw.wmnet with OS bullseye
* 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26875 and previous config saved to /var/cache/conftool/dbconfig/20220428-153523-ladsgroup.json
* 15:33 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 15:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2116.codfw.wmnet with OS bullseye
* 15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26874 and previous config saved to /var/cache/conftool/dbconfig/20220428-153307-ladsgroup.json
* 15:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 15:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26873 and previous config saved to /var/cache/conftool/dbconfig/20220428-153259-ladsgroup.json
* 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26872 and previous config saved to /var/cache/conftool/dbconfig/20220428-153233-ladsgroup.json
* 15:31 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2146.codfw.wmnet with OS bullseye
* 15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26871 and previous config saved to /var/cache/conftool/dbconfig/20220428-152924-ladsgroup.json
* 15:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 15:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26870 and previous config saved to /var/cache/conftool/dbconfig/20220428-152916-ladsgroup.json
* 15:27 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2092.codfw.wmnet with OS bullseye
* 15:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2072.codfw.wmnet with reason: host reimage
* 15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2130.codfw.wmnet with reason: host reimage
* 15:20 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup2002.codfw.wmnet
* 15:18 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2112.codfw.wmnet with reason: host reimage
* 15:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26869 and previous config saved to /var/cache/conftool/dbconfig/20220428-151754-ladsgroup.json
* 15:17 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2145.codfw.wmnet with reason: host reimage
* 15:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2116.codfw.wmnet with reason: host reimage
* 15:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2130.codfw.wmnet with reason: host reimage
* 15:14 kormat@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2146.codfw.wmnet with reason: host reimage
* 15:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2072.codfw.wmnet with reason: host reimage
* 15:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26868 and previous config saved to /var/cache/conftool/dbconfig/20220428-151411-ladsgroup.json
* 15:14 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2146.codfw.wmnet with reason: host reimage
* 15:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2145.codfw.wmnet with reason: host reimage
* 15:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2112.codfw.wmnet with reason: host reimage
* 15:11 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2092.codfw.wmnet with reason: host reimage
* 15:11 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2116.codfw.wmnet with reason: host reimage
* 15:08 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2092.codfw.wmnet with reason: host reimage
* 15:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26867 and previous config saved to /var/cache/conftool/dbconfig/20220428-150249-ladsgroup.json
* 15:00 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2072.codfw.wmnet with OS bullseye
* 15:00 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2146.codfw.wmnet with OS bullseye
* 14:59 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2145.codfw.wmnet with OS bullseye
* 14:59 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2130.codfw.wmnet with OS bullseye
* 14:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26866 and previous config saved to /var/cache/conftool/dbconfig/20220428-145906-ladsgroup.json
* 14:58 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2116.codfw.wmnet with OS bullseye
* 14:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:58 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS bullseye
* 14:58 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2092.codfw.wmnet with OS bullseye
* 14:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:58 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:56 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host restbase2025.codfw.wmnet
* 14:55 ladsgroup@deploy1002: Synchronized php-1.39.0-wmf.9/includes/specials/SpecialExport.php: Backport: [[gerrit:787089{{!}}SpecialExport: Add page table once (T307037)]] (duration: 00m 51s)
* 14:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2071.codfw.wmnet with OS bullseye
* 14:49 mforns@deploy1002: Finished deploy [airflow-dags/analytics@8278877]: (no justification provided) (duration: 00m 07s)
* 14:49 mforns@deploy1002: Started deploy [airflow-dags/analytics@8278877]: (no justification provided)
* 14:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26864 and previous config saved to /var/cache/conftool/dbconfig/20220428-144744-ladsgroup.json
* 14:46 moritzm: powercycling restbase2025
* 14:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26863 and previous config saved to /var/cache/conftool/dbconfig/20220428-144528-ladsgroup.json
* 14:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2002.codfw.wmnet with OS bullseye
* 14:44 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2072.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:44 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2072.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26862 and previous config saved to /var/cache/conftool/dbconfig/20220428-144401-ladsgroup.json
* 14:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2094.codfw.wmnet with reason: Reimaging db2072 [[phab:T303171|T303171]]
* 14:43 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2094.codfw.wmnet with reason: Reimaging db2072 [[phab:T303171|T303171]]
* 14:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26861 and previous config saved to /var/cache/conftool/dbconfig/20220428-144252-ladsgroup.json
* 14:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 14:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 14:42 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2146.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:42 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2146.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:41 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2145.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:41 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2145.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2116.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2116.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2130.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2130.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:40 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2112.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:40 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2112.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2025.codfw.wmnet
* 14:38 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2092.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:38 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2092.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2071.codfw.wmnet with reason: host reimage
* 14:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2002.codfw.wmnet with reason: host reimage
* 14:32 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db2071.codfw.wmnet with reason: host reimage
* 14:29 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2002.codfw.wmnet with reason: host reimage
* 14:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2024.codfw.wmnet
* 14:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2024.codfw.wmnet
* 14:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:17 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 14:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:15 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2002.codfw.wmnet with OS bullseye
* 14:14 kormat@cumin1001: START - Cookbook sre.hosts.reimage for host db2071.codfw.wmnet with OS bullseye
* 14:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:09 kormat: reimaging s1 to bullseye on s2@codfw  dbmaint ([[phab:T303171|T303171]])
* 14:08 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1013 as pc3 primary [[phab:T307101|T307101]] (duration: 00m 54s)
* 14:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2023.codfw.wmnet
* 14:07 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1004.eqiad.wmnet
* 14:06 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 14:06 Lucas_WMDE: UTC afternoon backport window done
* 14:06 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 14:05 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787111{{!}}itwiki: assign 'setmentor' to 'bot' usergroup (T307005)]] (duration: 00m 55s)
* 14:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 14:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 14:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 14:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 14:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:02 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2071.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:02 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw
* 14:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2071.codfw.wmnet with reason: Rebooting for [[phab:T303171|T303171]]
* 14:01 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/abusefilter.php: Config: [[gerrit:787115{{!}}zhwikiversity: Enable blocking feature of AbuseFilter (T307007)]] (duration: 00m 51s)
* 14:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:00 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1004.eqiad.wmnet
* 13:59 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2023.codfw.wmnet
* 13:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:52 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786407{{!}}Add kanoner.com to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T306795)]] (duration: 00m 50s)
* 13:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2023-2026].codfw.wmnet with reason: reboot
* 13:48 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2023-2026].codfw.wmnet with reason: reboot
* 13:45 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/logos.php: Config: [[gerrit:787114{{!}}zhwiki: Add comment to corresponding task of logo (T276694)]] (2/2, no-op) (duration: 00m 53s)
* 13:44 lucaswerkmeister-wmde@deploy1002: Synchronized logos/config.yaml: Config: [[gerrit:787114{{!}}zhwiki: Add comment to corresponding task of logo (T276694)]] (1/2, no-op) (duration: 00m 50s)
* 13:43 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1013.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:43 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1013.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:42 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Rebooting pc1013 [[phab:T307101|T307101]]
* 13:42 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2013.codfw.wmnet,pc[1013-1014].eqiad.wmnet with reason: Rebooting pc1013 [[phab:T307101|T307101]]
* 13:42 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc3 primary [[phab:T307101|T307101]] (duration: 00m 52s)
* 13:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2022.codfw.wmnet
* 13:37 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf 'https://en.wikipedia.org/static/images/project-logos/zhwiki-hans%s.png\n' '' '-1.5x' '-2x' {{!}} mwscript purgeList.php # [[phab:T276694|T276694]]
* 13:36 lucaswerkmeister-wmde@deploy1002: Synchronized static/images/project-logos/zhwiki-hans-2x.png: Config: [[gerrit:775320{{!}}Revert Simplified Chinese logo of zhwiki (T276694)]] (3/3) (duration: 00m 56s)
* 13:35 lucaswerkmeister-wmde@deploy1002: Synchronized static/images/project-logos/zhwiki-hans-1.5x.png: Config: [[gerrit:775320{{!}}Revert Simplified Chinese logo of zhwiki (T276694)]] (2/3) (duration: 00m 49s)
* 13:33 lucaswerkmeister-wmde@deploy1002: Synchronized static/images/project-logos/zhwiki-hans.png: Config: [[gerrit:775320{{!}}Revert Simplified Chinese logo of zhwiki (T276694)]] (1/3) (duration: 01m 25s)
* 13:32 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2022.codfw.wmnet
* 13:31 moritzm: drain ganeti-test2002 [[phab:T306499|T306499]]
* 13:29 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:22 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:22 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:15 elukey: upgrade scap to 4.7.1 to all nodes (except ores[12]* since they need to be upgraded to buster first)
* 13:15 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2021.codfw.wmnet
* 12:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2021.codfw.wmnet
* 12:54 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 12:54 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 12:53 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 12:53 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 12:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2020.codfw.wmnet
* 12:41 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2020.codfw.wmnet
* 12:39 moritzm: installing testvm2005 [[phab:T306499|T306499]]
* 12:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2020-2022].codfw.wmnet with reason: reboot
* 12:37 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2020-2022].codfw.wmnet with reason: reboot
* 12:32 krinkle@deploy1002: Synchronized w/static.php: {{Gerrit|I0bdf0b4038d8639858f6cb2ee90747d8c40e6294}} (duration: 01m 56s)
* 12:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2019.codfw.wmnet
* 12:28 jelto@cumin1001: conftool action : set/pooled=inactive; selector: name=mw1323.eqiad.wmnet
* 12:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2019.codfw.wmnet
* 12:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2019.codfw.wmnet with reason: reboot
* 12:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2019.codfw.wmnet with reason: reboot
* 12:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2018.codfw.wmnet
* 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2005.codfw.wmnet
* 11:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2018.codfw.wmnet
* 11:56 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 11:50 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw132[4-6].eqiad.wmnet
* 11:49 jelto: pool name=mw[1324-1326].eqiad.wmnet , manual puppet run and icinga green after reboot (cookbook failed because of mw1323 stuck in boot)
* 11:46 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 11:41 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2005.codfw.wmnet
* 11:40 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 11:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2017.codfw.wmnet
* 11:35 jynus: applying NIC firmware update onto backup2002 [[phab:T286722|T286722]]
* 11:33 moritzm: powercycling restbase2017
* 11:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2017.codfw.wmnet
* 11:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2016.codfw.wmnet
* 11:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2016.codfw.wmnet
* 11:01 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 10:56 moritzm: failover Ganeti master in ganeti-test to ganeti-test2001 (bullseye node) [[phab:T306499|T306499]]
* 10:56 moritzm: failover Ganeti master in ganeti-test to ganeti-test1001 (bullseye node) [[phab:T306499|T306499]]
* 10:55 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1012 as pc2 primary [[phab:T306983|T306983]] (duration: 00m 57s)
* 10:47 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 10:47 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1012.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:46 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1012.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:46 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 10:46 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 10:46 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc2 primary [[phab:T306983|T306983]] (duration: 00m 52s)
* 10:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Rebooting pc1012 [[phab:T306983|T306983]]
* 10:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Rebooting pc1012 [[phab:T306983|T306983]]
* 10:45 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
* 10:44 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 10:44 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
* 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2016-2018].codfw.wmnet with reason: reboot
* 10:43 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2016-2018].codfw.wmnet with reason: reboot
* 10:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2015.codfw.wmnet
* 10:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2015.codfw.wmnet
* 10:24 elukey@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 00m 18s)
* 10:24 elukey@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
* 10:23 elukey: update scap to 4.7.1 on restbase1016 (canary) - [[phab:T306998|T306998]]
* 10:20 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 10:15 elukey: update scap to 4.7.1 on A:mw-canary or A:parsoid-canary or A:mw-jobrunner-canary - [[phab:T306998|T306998]]
* 10:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2014.codfw.wmnet
* 10:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2014.codfw.wmnet
* 09:55 moritzm: failover idp.wikimedia.org to idp1001 for CAS update
* 09:51 marostegui: Change pc2014 innodb_max_dirty_pages_pct from 90% to 75% [[phab:T307082|T307082]]
* 09:41 marostegui: Change db1132 innodb_max_dirty_pages_pct from 90% to 75% [[phab:T307082|T307082]]
* 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2013.codfw.wmnet
* 09:32 moritzm: uploadded ganeti 3.0.1-2+deb11u0 to apt.wikimedia.org (backport of Py2->Py3 regression) [[phab:T306499|T306499]]
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2013.codfw.wmnet
* 09:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase[2013-2015].codfw.wmnet with reason: reboot
* 09:23 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on restbase[2013-2015].codfw.wmnet with reason: reboot
* 09:01 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host restbase2012.codfw.wmnet
* 08:58 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 08:58 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 08:58 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 08:57 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 08:56 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 08:55 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 08:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase2012.codfw.wmnet
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev1006.eqiad.wmnet
* 08:31 elukey: upload scap 4.7.1 to <nowiki>{</nowiki>buster,stretch,bullseye<nowiki>}</nowiki>-wikimedia apt repos - [[phab:T306998|T306998]]
* 08:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev1006.eqiad.wmnet
* 08:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev1005.eqiad.wmnet
* 08:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev1005.eqiad.wmnet
* 08:15 hashar: https://gerrit.wikimedia.org/ is now running 3.4.4 # [[phab:T292759|T292759]]
* 08:12 hashar: Stopping Gerrit for version ugprade # [[phab:T292759|T292759]]
* 08:10 hashar@deploy1002: Finished deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit1001 # [[phab:T292759|T292759]] (duration: 00m 09s)
* 08:10 hashar@deploy1002: Started deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit1001 # [[phab:T292759|T292759]]
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase-dev1004.eqiad.wmnet
* 08:04 hashar@deploy1002: Finished deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit2001 # [[phab:T292759|T292759]] (duration: 00m 11s)
* 08:04 hashar@deploy1002: Started deploy [gerrit/gerrit@031f315]: Gerrit to 3.4.4 on gerrit2001 # [[phab:T292759|T292759]]
* 08:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host restbase-dev1004.eqiad.wmnet
* 07:56 apergos: UTC morning backport and config training window closed
* 07:34 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 07:22 marostegui@cumin1001: dbctl commit (dc=all): 'Increase db1132 weight [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P26860 and previous config saved to /var/cache/conftool/dbconfig/20220428-072200-marostegui.json
* 07:19 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786947{{!}}Enable SectionTranslation in testwiki for Punjabi, Tsonga, Nepali, and Swahili (T304828)]] (duration: 00m 50s)
* 07:11 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786446{{!}}Enable Section Translation for Basque Wikipedia (T304862)]] (duration: 00m 50s)
* 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26859 and previous config saved to /var/cache/conftool/dbconfig/20220428-021733-ladsgroup.json
* 02:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26858 and previous config saved to /var/cache/conftool/dbconfig/20220428-020228-ladsgroup.json
* 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26857 and previous config saved to /var/cache/conftool/dbconfig/20220428-014723-ladsgroup.json
* 01:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26856 and previous config saved to /var/cache/conftool/dbconfig/20220428-013218-ladsgroup.json
* 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26855 and previous config saved to /var/cache/conftool/dbconfig/20220428-011007-ladsgroup.json
* 01:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 01:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26854 and previous config saved to /var/cache/conftool/dbconfig/20220428-010935-ladsgroup.json
* 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26853 and previous config saved to /var/cache/conftool/dbconfig/20220428-005430-ladsgroup.json
* 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26852 and previous config saved to /var/cache/conftool/dbconfig/20220428-003925-ladsgroup.json
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26851 and previous config saved to /var/cache/conftool/dbconfig/20220428-002420-ladsgroup.json
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26850 and previous config saved to /var/cache/conftool/dbconfig/20220428-002354-ladsgroup.json
* 00:15 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet
* 00:10 razzi@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet
* 00:09 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:09 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:09 razzi@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:09 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26849 and previous config saved to /var/cache/conftool/dbconfig/20220428-000849-ladsgroup.json
* 00:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26848 and previous config saved to /var/cache/conftool/dbconfig/20220428-000317-ladsgroup.json
* 00:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 00:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26847 and previous config saved to /var/cache/conftool/dbconfig/20220428-000244-ladsgroup.json
== 2022-04-27 ==
* 23:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26846 and previous config saved to /var/cache/conftool/dbconfig/20220427-235953-ladsgroup.json
* 23:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26845 and previous config saved to /var/cache/conftool/dbconfig/20220427-235700-ladsgroup.json
* 23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26844 and previous config saved to /var/cache/conftool/dbconfig/20220427-235344-ladsgroup.json
* 23:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26843 and previous config saved to /var/cache/conftool/dbconfig/20220427-234739-ladsgroup.json
* 23:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26842 and previous config saved to /var/cache/conftool/dbconfig/20220427-234448-ladsgroup.json
* 23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26841 and previous config saved to /var/cache/conftool/dbconfig/20220427-234155-ladsgroup.json
* 23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26840 and previous config saved to /var/cache/conftool/dbconfig/20220427-233839-ladsgroup.json
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26839 and previous config saved to /var/cache/conftool/dbconfig/20220427-233628-ladsgroup.json
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26838 and previous config saved to /var/cache/conftool/dbconfig/20220427-233541-ladsgroup.json
* 23:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26837 and previous config saved to /var/cache/conftool/dbconfig/20220427-233234-ladsgroup.json
* 23:31 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab-runner2002.codfw.wmnet with OS bullseye
* 23:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26836 and previous config saved to /var/cache/conftool/dbconfig/20220427-232942-ladsgroup.json
* 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26835 and previous config saved to /var/cache/conftool/dbconfig/20220427-232650-ladsgroup.json
* 23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26834 and previous config saved to /var/cache/conftool/dbconfig/20220427-232036-ladsgroup.json
* 23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26833 and previous config saved to /var/cache/conftool/dbconfig/20220427-231729-ladsgroup.json
* 23:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26832 and previous config saved to /var/cache/conftool/dbconfig/20220427-231437-ladsgroup.json
* 23:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26831 and previous config saved to /var/cache/conftool/dbconfig/20220427-231145-ladsgroup.json
* 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26830 and previous config saved to /var/cache/conftool/dbconfig/20220427-231037-ladsgroup.json
* 23:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26829 and previous config saved to /var/cache/conftool/dbconfig/20220427-231029-ladsgroup.json
* 23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26828 and previous config saved to /var/cache/conftool/dbconfig/20220427-230531-ladsgroup.json
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26827 and previous config saved to /var/cache/conftool/dbconfig/20220427-230130-ladsgroup.json
* 23:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26826 and previous config saved to /var/cache/conftool/dbconfig/20220427-230116-ladsgroup.json
* 22:59 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2002.codfw.wmnet with OS bullseye
* 22:58 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host gitlab2002.wikimedia.org with OS bullseye
* 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26825 and previous config saved to /var/cache/conftool/dbconfig/20220427-225524-ladsgroup.json
* 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26824 and previous config saved to /var/cache/conftool/dbconfig/20220427-225517-ladsgroup.json
* 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26823 and previous config saved to /var/cache/conftool/dbconfig/20220427-225445-ladsgroup.json
* 22:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26822 and previous config saved to /var/cache/conftool/dbconfig/20220427-225026-ladsgroup.json
* 22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26821 and previous config saved to /var/cache/conftool/dbconfig/20220427-224711-ladsgroup.json
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26820 and previous config saved to /var/cache/conftool/dbconfig/20220427-224654-ladsgroup.json
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26819 and previous config saved to /var/cache/conftool/dbconfig/20220427-224610-ladsgroup.json
* 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26818 and previous config saved to /var/cache/conftool/dbconfig/20220427-224019-ladsgroup.json
* 22:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26817 and previous config saved to /var/cache/conftool/dbconfig/20220427-223940-ladsgroup.json
* 22:39 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host gitlab2002.wikimedia.org with OS bullseye
* 22:34 mutante: kubernetes - Uprading release=namespaces/namspace-certificates which added developer-portal and image-suggestion namespaces - but only on staging-codfw - ([[phab:T304891|T304891]], [[phab:T305155|T305155]], [[phab:T297140|T297140]])
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26816 and previous config saved to /var/cache/conftool/dbconfig/20220427-223149-ladsgroup.json
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26815 and previous config saved to /var/cache/conftool/dbconfig/20220427-223105-ladsgroup.json
* 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26814 and previous config saved to /var/cache/conftool/dbconfig/20220427-222514-ladsgroup.json
* 22:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26813 and previous config saved to /var/cache/conftool/dbconfig/20220427-222435-ladsgroup.json
* 22:24 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
* 22:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26812 and previous config saved to /var/cache/conftool/dbconfig/20220427-222306-ladsgroup.json
* 22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26811 and previous config saved to /var/cache/conftool/dbconfig/20220427-222230-ladsgroup.json
* 22:19 dzahn@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 22:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2004.mgmt.codfw.wmnet with reboot policy FORCED
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26810 and previous config saved to /var/cache/conftool/dbconfig/20220427-221644-ladsgroup.json
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26809 and previous config saved to /var/cache/conftool/dbconfig/20220427-221600-ladsgroup.json
* 22:12 dzahn@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 22:09 mutante: running puppet on kubemasters - adding namespace to kubernetes for new service image-suggestion ([[phab:T304891|T304891]], [[phab:T305155|T305155]])
* 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26808 and previous config saved to /var/cache/conftool/dbconfig/20220427-220930-ladsgroup.json
* 22:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26807 and previous config saved to /var/cache/conftool/dbconfig/20220427-220725-ladsgroup.json
* 22:04 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26806 and previous config saved to /var/cache/conftool/dbconfig/20220427-220139-ladsgroup.json
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26805 and previous config saved to /var/cache/conftool/dbconfig/20220427-215914-ladsgroup.json
* 21:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26804 and previous config saved to /var/cache/conftool/dbconfig/20220427-215828-ladsgroup.json
* 21:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26803 and previous config saved to /var/cache/conftool/dbconfig/20220427-215820-ladsgroup.json
* 21:56 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 21:54 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26802 and previous config saved to /var/cache/conftool/dbconfig/20220427-215220-ladsgroup.json
* 21:50 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 21:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26801 and previous config saved to /var/cache/conftool/dbconfig/20220427-214825-ladsgroup.json
* 21:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 21:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26800 and previous config saved to /var/cache/conftool/dbconfig/20220427-214752-ladsgroup.json
* 21:46 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2412.codfw.wmnet
* 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26799 and previous config saved to /var/cache/conftool/dbconfig/20220427-214315-ladsgroup.json
* 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26798 and previous config saved to /var/cache/conftool/dbconfig/20220427-213715-ladsgroup.json
* 21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26797 and previous config saved to /var/cache/conftool/dbconfig/20220427-213507-ladsgroup.json
* 21:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26796 and previous config saved to /var/cache/conftool/dbconfig/20220427-213458-ladsgroup.json
* 21:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26795 and previous config saved to /var/cache/conftool/dbconfig/20220427-213247-ladsgroup.json
* 21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26794 and previous config saved to /var/cache/conftool/dbconfig/20220427-212810-ladsgroup.json
* 21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26793 and previous config saved to /var/cache/conftool/dbconfig/20220427-211953-ladsgroup.json
* 21:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26792 and previous config saved to /var/cache/conftool/dbconfig/20220427-211742-ladsgroup.json
* 21:16 ebernhardson@deploy1002: Synchronized wmf-config/ProductionServices.php: Config: Revert: [[gerrit:787068{{!}}cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 58s)
* 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26791 and previous config saved to /var/cache/conftool/dbconfig/20220427-211352-ladsgroup.json
* 21:13 ebernhardson@deploy1002: Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:787068{{!}}cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 54s)
* 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26790 and previous config saved to /var/cache/conftool/dbconfig/20220427-211305-ladsgroup.json
* 21:12 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787074{{!}}Forward CirrusSearchDeprecation logs to logstash (T218994)]] (duration: 00m 56s)
* 21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26789 and previous config saved to /var/cache/conftool/dbconfig/20220427-211055-ladsgroup.json
* 21:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26788 and previous config saved to /var/cache/conftool/dbconfig/20220427-211042-ladsgroup.json
* 21:09 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2004.mgmt.codfw.wmnet with reboot policy FORCED
* 21:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2003.mgmt.codfw.wmnet with reboot policy FORCED
* 21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26787 and previous config saved to /var/cache/conftool/dbconfig/20220427-210448-ladsgroup.json
* 21:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26786 and previous config saved to /var/cache/conftool/dbconfig/20220427-210237-ladsgroup.json
* 21:01 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787069{{!}}cirrus: Turn on AB test of wbsearchentities profiles (T306644)]] (duration: 00m 53s)
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26785 and previous config saved to /var/cache/conftool/dbconfig/20220427-210041-ladsgroup.json
* 21:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 21:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 20:59 dzahn@cumin2002: conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1009.eqiad.wmnet
* 20:56 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1009.eqiad.wmnet
* 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26784 and previous config saved to /var/cache/conftool/dbconfig/20220427-205537-ladsgroup.json
* 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26783 and previous config saved to /var/cache/conftool/dbconfig/20220427-204943-ladsgroup.json
* 20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26782 and previous config saved to /var/cache/conftool/dbconfig/20220427-204736-ladsgroup.json
* 20:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 20:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26781 and previous config saved to /var/cache/conftool/dbconfig/20220427-204728-ladsgroup.json
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26780 and previous config saved to /var/cache/conftool/dbconfig/20220427-204032-ladsgroup.json
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26779 and previous config saved to /var/cache/conftool/dbconfig/20220427-204031-ladsgroup.json
* 20:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 20:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26778 and previous config saved to /var/cache/conftool/dbconfig/20220427-203959-ladsgroup.json
* 20:39 mutante: alert1001 - systemctl start certspotter
* 20:36 mutante: ms-fe1009 - systemctl restart cron
* 20:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
* 20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
* 20:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26777 and previous config saved to /var/cache/conftool/dbconfig/20220427-203223-ladsgroup.json
* 20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26776 and previous config saved to /var/cache/conftool/dbconfig/20220427-202527-ladsgroup.json
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26775 and previous config saved to /var/cache/conftool/dbconfig/20220427-202454-ladsgroup.json
* 20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26774 and previous config saved to /var/cache/conftool/dbconfig/20220427-202314-ladsgroup.json
* 20:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:23 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2003.mgmt.codfw.wmnet with reboot policy FORCED
* 20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26773 and previous config saved to /var/cache/conftool/dbconfig/20220427-202306-ladsgroup.json
* 20:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26772 and previous config saved to /var/cache/conftool/dbconfig/20220427-202234-ladsgroup.json
* 20:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2002.mgmt.codfw.wmnet with reboot policy FORCED
* 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26771 and previous config saved to /var/cache/conftool/dbconfig/20220427-201718-ladsgroup.json
* 20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26770 and previous config saved to /var/cache/conftool/dbconfig/20220427-200949-ladsgroup.json
* 20:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26769 and previous config saved to /var/cache/conftool/dbconfig/20220427-200800-ladsgroup.json
* 20:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26768 and previous config saved to /var/cache/conftool/dbconfig/20220427-200729-ladsgroup.json
* 20:06 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784582{{!}}Halt the DiscussionTools A/B test (T291873)]] (duration: 00m 51s)
* 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26767 and previous config saved to /var/cache/conftool/dbconfig/20220427-200213-ladsgroup.json
* 20:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26766 and previous config saved to /var/cache/conftool/dbconfig/20220427-200006-ladsgroup.json
* 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26765 and previous config saved to /var/cache/conftool/dbconfig/20220427-195953-ladsgroup.json
* 19:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26764 and previous config saved to /var/cache/conftool/dbconfig/20220427-195444-ladsgroup.json
* 19:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26763 and previous config saved to /var/cache/conftool/dbconfig/20220427-195255-ladsgroup.json
* 19:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26762 and previous config saved to /var/cache/conftool/dbconfig/20220427-195224-ladsgroup.json
* 19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26761 and previous config saved to /var/cache/conftool/dbconfig/20220427-194448-ladsgroup.json
* 19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26760 and previous config saved to /var/cache/conftool/dbconfig/20220427-193749-ladsgroup.json
* 19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26759 and previous config saved to /var/cache/conftool/dbconfig/20220427-193719-ladsgroup.json
* 19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26758 and previous config saved to /var/cache/conftool/dbconfig/20220427-193233-ladsgroup.json
* 19:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 19:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26757 and previous config saved to /var/cache/conftool/dbconfig/20220427-193201-ladsgroup.json
* 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26756 and previous config saved to /var/cache/conftool/dbconfig/20220427-192943-ladsgroup.json
* 19:28 otto@deploy1002: Finished deploy [airflow-dags/analytics@6684963]: (no justification provided) (duration: 00m 09s)
* 19:28 otto@deploy1002: Started deploy [airflow-dags/analytics@6684963]: (no justification provided)
* 19:24 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 11s)
* 19:24 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 19:23 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 03s)
* 19:23 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26755 and previous config saved to /var/cache/conftool/dbconfig/20220427-192209-ladsgroup.json
* 19:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 19:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26754 and previous config saved to /var/cache/conftool/dbconfig/20220427-192200-ladsgroup.json
* 19:21 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 03s)
* 19:21 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 19:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2002.mgmt.codfw.wmnet with reboot policy FORCED
* 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26753 and previous config saved to /var/cache/conftool/dbconfig/20220427-191656-ladsgroup.json
* 19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26752 and previous config saved to /var/cache/conftool/dbconfig/20220427-191438-ladsgroup.json
* 19:08 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab2003.mgmt.codfw.wmnet with reboot policy FORCED
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26751 and previous config saved to /var/cache/conftool/dbconfig/20220427-190655-ladsgroup.json
* 19:02 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1004.eqiad.wmnet
* 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26750 and previous config saved to /var/cache/conftool/dbconfig/20220427-190151-ladsgroup.json
* 19:00 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:00 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26749 and previous config saved to /var/cache/conftool/dbconfig/20220427-185150-ladsgroup.json
* 18:51 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1004.eqiad.wmnet
* 18:50 dancy@deploy1002: Started scap: testing mediawiki container image build and deploy
* 18:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26748 and previous config saved to /var/cache/conftool/dbconfig/20220427-184646-ladsgroup.json
* 18:46 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 12s)
* 18:46 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 18:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab2003.mgmt.codfw.wmnet with reboot policy FORCED
* 18:45 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:45 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:38 dancy@deploy1002: Started scap: (no justification provided)
* 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26746 and previous config saved to /var/cache/conftool/dbconfig/20220427-183735-ladsgroup.json
* 18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26745 and previous config saved to /var/cache/conftool/dbconfig/20220427-183727-ladsgroup.json
* 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26744 and previous config saved to /var/cache/conftool/dbconfig/20220427-183645-ladsgroup.json
* 18:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab2002.mgmt.codfw.wmnet with reboot policy FORCED
* 18:26 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1003.eqiad.wmnet
* 18:25 brennen@deploy1002: Synchronized php: group1 wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]] (duration: 00m 56s)
* 18:24 brennen@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]]
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26743 and previous config saved to /var/cache/conftool/dbconfig/20220427-182238-ladsgroup.json
* 18:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 18:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26742 and previous config saved to /var/cache/conftool/dbconfig/20220427-182230-ladsgroup.json
* 18:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26741 and previous config saved to /var/cache/conftool/dbconfig/20220427-182226-ladsgroup.json
* 18:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26740 and previous config saved to /var/cache/conftool/dbconfig/20220427-182222-ladsgroup.json
* 18:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 18:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 18:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26739 and previous config saved to /var/cache/conftool/dbconfig/20220427-181423-ladsgroup.json
* 18:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 18:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26738 and previous config saved to /var/cache/conftool/dbconfig/20220427-181415-ladsgroup.json
* 18:14 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1003.eqiad.wmnet
* 18:12 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab2002.mgmt.codfw.wmnet with reboot policy FORCED
* 18:11 brennen@deploy1002: Synchronized php-1.39.0-wmf.9/skins/WikimediaApiPortal: Backport: [[gerrit:786420{{!}}Fix warnings relating to QuickTemplate (T306925)]] (duration: 02m 26s)
* 18:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26737 and previous config saved to /var/cache/conftool/dbconfig/20220427-180725-ladsgroup.json
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26736 and previous config saved to /var/cache/conftool/dbconfig/20220427-180717-ladsgroup.json
* 18:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:00 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers, proceeding to group1
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26735 and previous config saved to /var/cache/conftool/dbconfig/20220427-175910-ladsgroup.json
* 17:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26734 and previous config saved to /var/cache/conftool/dbconfig/20220427-175220-ladsgroup.json
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26733 and previous config saved to /var/cache/conftool/dbconfig/20220427-175212-ladsgroup.json
* 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26732 and previous config saved to /var/cache/conftool/dbconfig/20220427-175100-ladsgroup.json
* 17:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 17:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 17:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:49 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:49 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:49 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:45 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26731 and previous config saved to /var/cache/conftool/dbconfig/20220427-174405-ladsgroup.json
* 17:41 ejegg: updated payments-wiki from {{Gerrit|16c0c111}} to {{Gerrit|41e91033}}
* 17:39 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1002.eqiad.wmnet
* 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26730 and previous config saved to /var/cache/conftool/dbconfig/20220427-173709-ladsgroup.json
* 17:36 dancy: dancy@deploy1002  Testing image build and deployment
* 17:35 dancy@deploy1002: Started scap: (no justification provided)
* 17:29 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1002.eqiad.wmnet
* 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26729 and previous config saved to /var/cache/conftool/dbconfig/20220427-172900-ladsgroup.json
* 17:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|88ac6b94ff5acfbcd6a539e21b7454b6a38eaaba}}: Enable Wikistories on enwiki beta ([[phab:T303004|T303004]]; 2/2) (duration: 00m 50s)
* 17:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|88ac6b94ff5acfbcd6a539e21b7454b6a38eaaba}}: Enable Wikistories on enwiki beta ([[phab:T303004|T303004]]; 1/2) (duration: 00m 51s)
* 17:27 urbanecm@deploy1002: Synchronized wmf-config/extension-list: {{Gerrit|01dfaf063d14ee329c43d65566270ff3cec48d50}}: Add Wikistories extension ([[phab:T303004|T303004]]) (duration: 00m 49s)
* 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26728 and previous config saved to /var/cache/conftool/dbconfig/20220427-172653-ladsgroup.json
* 17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 17:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 17:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26727 and previous config saved to /var/cache/conftool/dbconfig/20220427-172211-ladsgroup.json
* 17:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:10 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 17:01 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:58 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 16:56 herron@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host thanos-be1001.eqiad.wmnet
* 16:53 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 16:52 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 16:52 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 16:46 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 16:43 dancy@deploy1002: Finished deploy [releng/phatality@d8e2adc]: (no justification provided) (duration: 00m 13s)
* 16:43 dancy@deploy1002: Started deploy [releng/phatality@d8e2adc]: (no justification provided)
* 16:41 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1001.eqiad.wmnet
* 16:35 razzi@cumin1001: conftool action : set/pooled=inactive; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 16:34 razzi@cumin1001: conftool action : set/pooled=yes; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 16:28 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26725 and previous config saved to /var/cache/conftool/dbconfig/20220427-161459-ladsgroup.json
* 16:04 Amir1: foreachwikiindblist s6 mysql.php -- -e "desc querycache_info;" {{!}} grep -i qci_timestamp {{!}} grep -i varbinary {{!}} awk '<nowiki>{</nowiki> print substr($1, 1, length($1)-1) <nowiki>}</nowiki>' {{!}} xargs -I <nowiki>{</nowiki><nowiki>}</nowiki> sql <nowiki>{</nowiki><nowiki>}</nowiki> --write -- -e 'ALTER TABLE /*_*/querycache_info CHANGE qci_timestamp qci_timestamp BINARY(14) DEFAULT '19700101000000' NOT NULL;' ([[phab:T298559|T298559]])
* 15:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26724 and previous config saved to /var/cache/conftool/dbconfig/20220427-155954-ladsgroup.json
* 15:58 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1003.eqiad.wmnet
* 15:56 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 15:56 jayme@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 15:52 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2004.codfw.wmnet
* 15:50 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1003.eqiad.wmnet
* 15:50 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet
* 15:48 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet
* 15:47 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet
* 15:45 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet
* 15:45 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@6684963]: (no justification provided) (duration: 00m 08s)
* 15:45 nokafor@deploy1002: Started deploy [airflow-dags/analytics@6684963]: (no justification provided)
* 15:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26723 and previous config saved to /var/cache/conftool/dbconfig/20220427-154449-ladsgroup.json
* 15:39 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2004.codfw.wmnet
* 15:36 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 15:35 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 15:33 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 15:31 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26722 and previous config saved to /var/cache/conftool/dbconfig/20220427-152944-ladsgroup.json
* 15:02 moritzm: installing mariadb-10.5 updates (as packaged in Debian Bullseye, unrelated to wmf-mariadb packages)
* 14:57 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2003.codfw.wmnet
* 14:54 razzi@cumin1001: conftool action : set/pooled=inactive; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 14:54 razzi@cumin1001: conftool action : set/pooled=no; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 14:53 razzi@cumin1001: conftool action : set/pooled=yes; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 14:47 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2003.codfw.wmnet
* 14:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:43 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786976{{!}}Enable videojs in eswiki (T303785 T248418)]] (duration: 00m 51s)
* 14:42 moritzm: imported cas 6.4.6.3 to apt.wikimedia.org
* 14:27 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 14:23 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 14:22 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2002.codfw.wmnet
* 14:21 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26720 and previous config saved to /var/cache/conftool/dbconfig/20220427-141215-ladsgroup.json
* 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 14:09 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2002.codfw.wmnet
* 14:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 14:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 14:07 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6001.drmrs.wmnet
* 14:03 klausman@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 14:03 klausman@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 14:00 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus6001.drmrs.wmnet
* 13:59 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4001.ulsfo.wmnet
* 13:54 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus4001.ulsfo.wmnet
* 13:53 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5001.eqsin.wmnet
* 13:51 jayme@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:47 jayme@cumin1001: START - Cookbook sre.dns.netbox
* 13:46 moritzm: rebalance ganeti-test after adding new bullseye node  [[phab:T306499|T306499]]
* 13:46 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus5001.eqsin.wmnet
* 13:45 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3001.esams.wmnet
* 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26719 and previous config saved to /var/cache/conftool/dbconfig/20220427-134308-ladsgroup.json
* 13:40 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus3001.esams.wmnet
* 13:37 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
* 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26718 and previous config saved to /var/cache/conftool/dbconfig/20220427-133537-ladsgroup.json
* 13:35 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26717 and previous config saved to /var/cache/conftool/dbconfig/20220427-132802-ladsgroup.json
* 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26716 and previous config saved to /var/cache/conftool/dbconfig/20220427-132032-ladsgroup.json
* 13:13 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 13:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26715 and previous config saved to /var/cache/conftool/dbconfig/20220427-131257-ladsgroup.json
* 13:11 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
* 13:05 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26714 and previous config saved to /var/cache/conftool/dbconfig/20220427-130527-ladsgroup.json
* 12:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26712 and previous config saved to /var/cache/conftool/dbconfig/20220427-125752-ladsgroup.json
* 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26710 and previous config saved to /var/cache/conftool/dbconfig/20220427-125427-ladsgroup.json
* 12:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 12:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 12:53 kormat: moved pc1014 to pc2 [[phab:T303174|T303174]]
* 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26709 and previous config saved to /var/cache/conftool/dbconfig/20220427-125022-ladsgroup.json
* 12:47 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26708 and previous config saved to /var/cache/conftool/dbconfig/20220427-124714-ladsgroup.json
* 12:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 12:42 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on eqiad
* 12:42 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2010.codfw.wmnet with OS bullseye
* 12:28 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage
* 12:25 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on drmrs
* 12:25 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage
* 12:15 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on esams
* 12:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 12:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 12:07 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS bullseye
* 12:07 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786936{{!}}Set actor migration to READ NEW everywhere (T275246)]] (duration: 00m 53s)
* 11:58 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2011.codfw.wmnet with OS bullseye
* 11:44 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage
* 11:42 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 11:41 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage
* 11:24 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS bullseye
* 11:15 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2012.codfw.wmnet with OS bullseye
* 11:02 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage
* 10:58 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage
* 10:51 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on eqsin
* 10:40 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2012.codfw.wmnet with OS bullseye
* 10:25 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on codfw
* 10:03 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1010.eqiad.wmnet with OS bullseye
* 09:50 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage
* 09:47 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage
* 09:44 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on ulsfo
* 09:33 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe1010.eqiad.wmnet with OS bullseye
* 09:30 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1446.eqiad.wmnet
* 09:30 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1445.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1440.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1439.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1438.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1437.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1338.eqiad.wmnet
* 09:29 akosiaris: repool the hosts we split off on 2022-04-23 as dedicated videoscalers in the jobrunner cluster. The videoscaling load seems to be normal again.
* 09:22 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1011.eqiad.wmnet with OS bullseye
* 09:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage
* 09:08 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage
* 08:54 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe1011.eqiad.wmnet with OS bullseye
* 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26706 and previous config saved to /var/cache/conftool/dbconfig/20220427-084941-ladsgroup.json
* 08:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26705 and previous config saved to /var/cache/conftool/dbconfig/20220427-083436-ladsgroup.json
* 08:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26704 and previous config saved to /var/cache/conftool/dbconfig/20220427-081931-ladsgroup.json
* 08:17 marostegui@cumin1001: dbctl commit (dc=all): 'Increase db1132 weight [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P26703 and previous config saved to /var/cache/conftool/dbconfig/20220427-081727-marostegui.json
* 08:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26702 and previous config saved to /var/cache/conftool/dbconfig/20220427-080425-ladsgroup.json
* 07:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti-test2001.codfw.wmnet with reason: bullseye update
* 07:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti-test2001.codfw.wmnet with reason: bullseye update
* 07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26701 and previous config saved to /var/cache/conftool/dbconfig/20220427-073114-ladsgroup.json
* 07:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 07:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26700 and previous config saved to /var/cache/conftool/dbconfig/20220427-073106-ladsgroup.json
* 07:24 moritzm: installing libxml2 security updates
* 07:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26699 and previous config saved to /var/cache/conftool/dbconfig/20220427-071601-ladsgroup.json
* 07:11 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 07:11 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26698 and previous config saved to /var/cache/conftool/dbconfig/20220427-070056-ladsgroup.json
* 06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26697 and previous config saved to /var/cache/conftool/dbconfig/20220427-064551-ladsgroup.json
* 06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26696 and previous config saved to /var/cache/conftool/dbconfig/20220427-061112-ladsgroup.json
* 06:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 06:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26695 and previous config saved to /var/cache/conftool/dbconfig/20220427-061104-ladsgroup.json
* 05:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26694 and previous config saved to /var/cache/conftool/dbconfig/20220427-055559-ladsgroup.json
* 05:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26693 and previous config saved to /var/cache/conftool/dbconfig/20220427-054054-ladsgroup.json
* 05:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26692 and previous config saved to /var/cache/conftool/dbconfig/20220427-052555-ladsgroup.json
* 05:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26691 and previous config saved to /var/cache/conftool/dbconfig/20220427-052549-ladsgroup.json
* 05:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26690 and previous config saved to /var/cache/conftool/dbconfig/20220427-051050-ladsgroup.json
* 04:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26689 and previous config saved to /var/cache/conftool/dbconfig/20220427-045545-ladsgroup.json
* 04:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26688 and previous config saved to /var/cache/conftool/dbconfig/20220427-045153-ladsgroup.json
* 04:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 04:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26687 and previous config saved to /var/cache/conftool/dbconfig/20220427-044040-ladsgroup.json
* 04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26686 and previous config saved to /var/cache/conftool/dbconfig/20220427-043732-ladsgroup.json
* 04:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 04:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26685 and previous config saved to /var/cache/conftool/dbconfig/20220427-043724-ladsgroup.json
* 04:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26684 and previous config saved to /var/cache/conftool/dbconfig/20220427-042219-ladsgroup.json
* 04:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 04:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 04:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26683 and previous config saved to /var/cache/conftool/dbconfig/20220427-040714-ladsgroup.json
* 04:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 04:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 03:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 03:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26682 and previous config saved to /var/cache/conftool/dbconfig/20220427-035208-ladsgroup.json
* 03:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26681 and previous config saved to /var/cache/conftool/dbconfig/20220427-035001-ladsgroup.json
* 03:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 03:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 03:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 03:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 03:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26680 and previous config saved to /var/cache/conftool/dbconfig/20220427-034948-ladsgroup.json
* 03:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26679 and previous config saved to /var/cache/conftool/dbconfig/20220427-033443-ladsgroup.json
* 03:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 03:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26678 and previous config saved to /var/cache/conftool/dbconfig/20220427-031938-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26677 and previous config saved to /var/cache/conftool/dbconfig/20220427-030433-ladsgroup.json
* 03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26676 and previous config saved to /var/cache/conftool/dbconfig/20220427-030225-ladsgroup.json
* 03:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 03:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26675 and previous config saved to /var/cache/conftool/dbconfig/20220427-030217-ladsgroup.json
* 02:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26674 and previous config saved to /var/cache/conftool/dbconfig/20220427-024712-ladsgroup.json
* 02:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26673 and previous config saved to /var/cache/conftool/dbconfig/20220427-024409-ladsgroup.json
* 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26672 and previous config saved to /var/cache/conftool/dbconfig/20220427-023207-ladsgroup.json
* 02:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26671 and previous config saved to /var/cache/conftool/dbconfig/20220427-022053-ladsgroup.json
* 02:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 02:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 02:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26670 and previous config saved to /var/cache/conftool/dbconfig/20220427-022045-ladsgroup.json
* 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26669 and previous config saved to /var/cache/conftool/dbconfig/20220427-021702-ladsgroup.json
* 02:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26668 and previous config saved to /var/cache/conftool/dbconfig/20220427-021450-ladsgroup.json
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26667 and previous config saved to /var/cache/conftool/dbconfig/20220427-021405-ladsgroup.json
* 02:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26666 and previous config saved to /var/cache/conftool/dbconfig/20220427-020540-ladsgroup.json
* 01:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26665 and previous config saved to /var/cache/conftool/dbconfig/20220427-015900-ladsgroup.json
* 01:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26664 and previous config saved to /var/cache/conftool/dbconfig/20220427-015035-ladsgroup.json
* 01:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26663 and previous config saved to /var/cache/conftool/dbconfig/20220427-014355-ladsgroup.json
* 01:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26662 and previous config saved to /var/cache/conftool/dbconfig/20220427-013530-ladsgroup.json
* 01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26661 and previous config saved to /var/cache/conftool/dbconfig/20220427-012850-ladsgroup.json
* 01:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26660 and previous config saved to /var/cache/conftool/dbconfig/20220427-012538-ladsgroup.json
* 01:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26659 and previous config saved to /var/cache/conftool/dbconfig/20220427-012530-ladsgroup.json
* 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26658 and previous config saved to /var/cache/conftool/dbconfig/20220427-011025-ladsgroup.json
* 01:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26657 and previous config saved to /var/cache/conftool/dbconfig/20220427-010001-ladsgroup.json
* 01:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 01:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26656 and previous config saved to /var/cache/conftool/dbconfig/20220427-005953-ladsgroup.json
* 00:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26655 and previous config saved to /var/cache/conftool/dbconfig/20220427-005520-ladsgroup.json
* 00:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26654 and previous config saved to /var/cache/conftool/dbconfig/20220427-004448-ladsgroup.json
* 00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26653 and previous config saved to /var/cache/conftool/dbconfig/20220427-004015-ladsgroup.json
* 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26652 and previous config saved to /var/cache/conftool/dbconfig/20220427-002943-ladsgroup.json
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26651 and previous config saved to /var/cache/conftool/dbconfig/20220427-002432-ladsgroup.json
* 00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26650 and previous config saved to /var/cache/conftool/dbconfig/20220427-001438-ladsgroup.json
* 00:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26649 and previous config saved to /var/cache/conftool/dbconfig/20220427-000927-ladsgroup.json
== 2022-04-26 ==
* 23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26648 and previous config saved to /var/cache/conftool/dbconfig/20220426-235422-ladsgroup.json
* 23:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26647 and previous config saved to /var/cache/conftool/dbconfig/20220426-234224-ladsgroup.json
* 23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26646 and previous config saved to /var/cache/conftool/dbconfig/20220426-234000-ladsgroup.json
* 23:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 23:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26645 and previous config saved to /var/cache/conftool/dbconfig/20220426-233953-ladsgroup.json
* 23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26644 and previous config saved to /var/cache/conftool/dbconfig/20220426-233917-ladsgroup.json
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26643 and previous config saved to /var/cache/conftool/dbconfig/20220426-233642-ladsgroup.json
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 14 hosts with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 14 hosts with reason: Maintenance
* 23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26642 and previous config saved to /var/cache/conftool/dbconfig/20220426-233545-ladsgroup.json
* 23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26641 and previous config saved to /var/cache/conftool/dbconfig/20220426-232447-ladsgroup.json
* 23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26640 and previous config saved to /var/cache/conftool/dbconfig/20220426-232040-ladsgroup.json
* 23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26639 and previous config saved to /var/cache/conftool/dbconfig/20220426-230942-ladsgroup.json
* 23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26638 and previous config saved to /var/cache/conftool/dbconfig/20220426-230535-ladsgroup.json
* 22:57 tgr@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/extension.json: Backport: [[gerrit:786366{{!}}Enable SkinAddFooterLinks hook]] (duration: 00m 51s)
* 22:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26637 and previous config saved to /var/cache/conftool/dbconfig/20220426-225437-ladsgroup.json
* 22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26636 and previous config saved to /var/cache/conftool/dbconfig/20220426-225326-ladsgroup.json
* 22:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26635 and previous config saved to /var/cache/conftool/dbconfig/20220426-225030-ladsgroup.json
* 22:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26634 and previous config saved to /var/cache/conftool/dbconfig/20220426-224757-ladsgroup.json
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26633 and previous config saved to /var/cache/conftool/dbconfig/20220426-224749-ladsgroup.json
* 22:46 tgr@deploy1002: Finished scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]] (duration: 21m 40s)
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26632 and previous config saved to /var/cache/conftool/dbconfig/20220426-223244-ladsgroup.json
* 22:25 tgr@deploy1002: Started scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]]
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26631 and previous config saved to /var/cache/conftool/dbconfig/20220426-221739-ladsgroup.json
* 22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26630 and previous config saved to /var/cache/conftool/dbconfig/20220426-220234-ladsgroup.json
* 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26629 and previous config saved to /var/cache/conftool/dbconfig/20220426-220001-ladsgroup.json
* 22:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 22:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26628 and previous config saved to /var/cache/conftool/dbconfig/20220426-215953-ladsgroup.json
* 21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26627 and previous config saved to /var/cache/conftool/dbconfig/20220426-214448-ladsgroup.json
* 21:38 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9] (duration: 00m 07s)
* 21:37 aqu@deploy1002: Started deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9]
* 21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26626 and previous config saved to /var/cache/conftool/dbconfig/20220426-212943-ladsgroup.json
* 21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26625 and previous config saved to /var/cache/conftool/dbconfig/20220426-211437-ladsgroup.json
* 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26624 and previous config saved to /var/cache/conftool/dbconfig/20220426-211204-ladsgroup.json
* 21:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26623 and previous config saved to /var/cache/conftool/dbconfig/20220426-211156-ladsgroup.json
* 21:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2418.codfw.wmnet
* 21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2417.codfw.wmnet
* 21:02 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2416.codfw.wmnet
* 21:01 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|cab00628da0ba6226ff162cfc848bea35a35783a}}: fix wmgVectorMaxWidthOptionsNamespaces ([[phab:T300182|T300182]]) (duration: 01m 00s)
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:00 mutante: mw2416, mw2417, mw2418 - scap pull
* 20:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26621 and previous config saved to /var/cache/conftool/dbconfig/20220426-205651-ladsgroup.json
* 20:50 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 07s)
* 20:50 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
* 20:49 urbanecm@deploy1002: Synchronized wmf-config/SearchSettingsForWikidata.php: {{Gerrit|f76bc806157a3f4c88d44cd467de347b4b471f4e}}: Correct wbsearchentities profiles (duration: 00m 57s)
* 20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26620 and previous config saved to /var/cache/conftool/dbconfig/20220426-204146-ladsgroup.json
* 20:40 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2415.codfw.wmnet
* 20:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2414.codfw.wmnet
* 20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2413.codfw.wmnet
* 20:38 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2412.codfw.wmnet
* 20:37 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.9/skins/Vector/resources/skins.vector.styles/: {{Gerrit|019a812176bb940383ddeb22f8a74b5d0f447bf1}}: [ToC] Increase threshold for ToC collapsing to 1000px ([[phab:T306904|T306904]]) (duration: 00m 50s)
* 20:36 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/skins/Vector/resources/skins.vector.styles/: {{Gerrit|31ed884d6eda998f8625a88be0f4aa5fd67aef4b}}: [ToC] Increase threshold for ToC collapsing to 1000px ([[phab:T306904|T306904]]) (duration: 00m 50s)
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:33 mutante: mw2412, mw2413, mw2414, mw2415 - scap pull, get into production the first time
* 20:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|fe0e119ef7c768373db4afed21537f85004a8ae2}}: Expand max-width to login, create account, disable on Wikidata ([[phab:T300182|T300182]], [[phab:T306834|T306834]]; 2/2) (duration: 00m 54s)
* 20:28 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 17s)
* 20:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|fe0e119ef7c768373db4afed21537f85004a8ae2}}: Expand max-width to login, create account, disable on Wikidata ([[phab:T300182|T300182]], [[phab:T306834|T306834]]; 1/2) (duration: 00m 56s)
* 20:27 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
* 20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26619 and previous config saved to /var/cache/conftool/dbconfig/20220426-202641-ladsgroup.json
* 20:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|e3ce97b97c1d83dc4f538040da92a571895cb4d0}}: Enable table of contents a/b test on euwiki and hewiki, enable reading depth ([[phab:T306606|T306606]]) (duration: 00m 52s)
* 20:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26618 and previous config saved to /var/cache/conftool/dbconfig/20220426-202407-ladsgroup.json
* 20:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 20:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26617 and previous config saved to /var/cache/conftool/dbconfig/20220426-202359-ladsgroup.json
* 20:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26616 and previous config saved to /var/cache/conftool/dbconfig/20220426-201610-ladsgroup.json
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:11 urbanecm@deploy1002: Synchronized wmf-config/: {{Gerrit|9805e61f7006edf45199a3e22494945bffaaeb4d}}: Add wbsearchentities profiles for testing ([[phab:T306644|T306644]]) (duration: 00m 53s)
* 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26615 and previous config saved to /var/cache/conftool/dbconfig/20220426-200854-ladsgroup.json
* 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:05 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|080b8fc573d9d682038e09a7a7ad875bce478c00}}: cirrus: Turn on retry_on_conflict quirk (duration: 00m 53s)
* 20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26614 and previous config saved to /var/cache/conftool/dbconfig/20220426-200105-ladsgroup.json
* 19:54 dzahn@cumin2002: conftool action : set/pooled=yes; selector: dc=codfw,name=mw2419.codfw.wmnet
* 19:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26612 and previous config saved to /var/cache/conftool/dbconfig/20220426-195349-ladsgroup.json
* 19:48 mutante: mw2419 - set weight to 25 in conftool, scap pull, first time in production, jobrunner/videoscaler [[phab:T290192|T290192]]
* 19:46 dzahn@cumin2002: conftool action : set/weight=25; selector: dc=codfw,name=mw2419.codfw.wmnet
* 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26611 and previous config saved to /var/cache/conftool/dbconfig/20220426-194600-ladsgroup.json
* 19:45 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2419.codfw.wmnet
* 19:42 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96a3934] (duration: 07m 19s)
* 19:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26610 and previous config saved to /var/cache/conftool/dbconfig/20220426-193844-ladsgroup.json
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1164 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26609 and previous config saved to /var/cache/conftool/dbconfig/20220426-193610-ladsgroup.json
* 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26608 and previous config saved to /var/cache/conftool/dbconfig/20220426-193602-ladsgroup.json
* 19:34 aqu@deploy1002: Started deploy [analytics/refinery@96a3934] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96a3934]
* 19:34 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934] (thin): Regular analytics weekly train THIN [analytics/refinery@96a3934] (duration: 00m 07s)
* 19:34 aqu@deploy1002: Started deploy [analytics/refinery@96a3934] (thin): Regular analytics weekly train THIN [analytics/refinery@96a3934]
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26607 and previous config saved to /var/cache/conftool/dbconfig/20220426-193055-ladsgroup.json
* 19:30 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934]: Regular analytics weekly train [analytics/refinery@96a3934] (duration: 24m 35s)
* 19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26606 and previous config saved to /var/cache/conftool/dbconfig/20220426-192841-ladsgroup.json
* 19:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 19:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 19:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 19:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 19:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26605 and previous config saved to /var/cache/conftool/dbconfig/20220426-192828-ladsgroup.json
* 19:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P26604 and previous config saved to /var/cache/conftool/dbconfig/20220426-192057-ladsgroup.json
* 19:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26603 and previous config saved to /var/cache/conftool/dbconfig/20220426-191323-ladsgroup.json
* 19:06 aqu@deploy1002: Started deploy [analytics/refinery@96a3934]: Regular analytics weekly train [analytics/refinery@96a3934]
* 19:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P26602 and previous config saved to /var/cache/conftool/dbconfig/20220426-190552-ladsgroup.json
* 19:02 aqu: About to deploy analytics/refinery: Weekly deployment train + Artifacts to 0.1.27
* 18:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26601 and previous config saved to /var/cache/conftool/dbconfig/20220426-185818-ladsgroup.json
* 18:50 cmooney@cumin1001: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1a - cmooney@cumin1001
* 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26599 and previous config saved to /var/cache/conftool/dbconfig/20220426-185047-ladsgroup.json
* 18:49 cmooney@cumin1001: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1a - cmooney@cumin1001
* 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26598 and previous config saved to /var/cache/conftool/dbconfig/20220426-184815-ladsgroup.json
* 18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 18:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 18:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
* 18:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
* 18:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26597 and previous config saved to /var/cache/conftool/dbconfig/20220426-184729-ladsgroup.json
* 18:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26596 and previous config saved to /var/cache/conftool/dbconfig/20220426-184313-ladsgroup.json
* 18:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26595 and previous config saved to /var/cache/conftool/dbconfig/20220426-184058-ladsgroup.json
* 18:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 18:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 18:37 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P26594 and previous config saved to /var/cache/conftool/dbconfig/20220426-183224-ladsgroup.json
* 18:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P26593 and previous config saved to /var/cache/conftool/dbconfig/20220426-181719-ladsgroup.json
* 18:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:06 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]]
* 18:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1132 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26592 and previous config saved to /var/cache/conftool/dbconfig/20220426-180214-ladsgroup.json
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1132 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26591 and previous config saved to /var/cache/conftool/dbconfig/20220426-175941-ladsgroup.json
* 17:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
* 17:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26590 and previous config saved to /var/cache/conftool/dbconfig/20220426-175933-ladsgroup.json
* 17:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26589 and previous config saved to /var/cache/conftool/dbconfig/20220426-175536-ladsgroup.json
* 17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26588 and previous config saved to /var/cache/conftool/dbconfig/20220426-175424-ladsgroup.json
* 17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance
* 17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance
* 17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 17:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 17:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 17:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26587 and previous config saved to /var/cache/conftool/dbconfig/20220426-175322-ladsgroup.json
* 17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P26586 and previous config saved to /var/cache/conftool/dbconfig/20220426-174428-ladsgroup.json
* 17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26585 and previous config saved to /var/cache/conftool/dbconfig/20220426-173817-ladsgroup.json
* 17:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:30 brennen@deploy1002: Pruned MediaWiki: 1.39.0-wmf.7 (duration: 01m 29s)
* 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P26584 and previous config saved to /var/cache/conftool/dbconfig/20220426-172923-ladsgroup.json
* 17:28 brennen@deploy1002: Finished scap: Re-running sync-world to see if timeouts recur for 32 hosts ([[phab:T305215|T305215]]) (duration: 01m 43s)
* 17:26 brennen@deploy1002: Started scap: Re-running sync-world to see if timeouts recur for 32 hosts ([[phab:T305215|T305215]])
* 17:23 mutante: mw2309 - scap pull
* 17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26583 and previous config saved to /var/cache/conftool/dbconfig/20220426-172312-ladsgroup.json
* 17:23 brennen@deploy1002: Finished scap: testwikis wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]] (duration: 34m 37s)
* 17:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:15 mutante: wtp1046 - scap pull
* 17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26582 and previous config saved to /var/cache/conftool/dbconfig/20220426-171418-ladsgroup.json
* 17:13 mutante: mw1362 - scap pull
* 17:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26580 and previous config saved to /var/cache/conftool/dbconfig/20220426-171144-ladsgroup.json
* 17:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 17:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 17:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 17:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 17:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 17:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 17:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26579 and previous config saved to /var/cache/conftool/dbconfig/20220426-171032-ladsgroup.json
* 17:09 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1021.eqiad.wmnet with OS bullseye
* 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26578 and previous config saved to /var/cache/conftool/dbconfig/20220426-170807-ladsgroup.json
* 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26577 and previous config saved to /var/cache/conftool/dbconfig/20220426-170553-ladsgroup.json
* 17:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 17:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 17:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26576 and previous config saved to /var/cache/conftool/dbconfig/20220426-170545-ladsgroup.json
* 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P26575 and previous config saved to /var/cache/conftool/dbconfig/20220426-165526-ladsgroup.json
* 16:55 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage
* 16:51 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb1021.eqiad.wmnet with reason: host reimage
* 16:50 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26574 and previous config saved to /var/cache/conftool/dbconfig/20220426-165040-ladsgroup.json
* 16:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:48 brennen@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]]
* 16:47 brennen: forgot SCAP=scap environment variable, re-running testwiki sync
* 16:46 brennen@deploy1002: stage-train aborted:  (duration: 06m 04s)
* 16:46 brennen@deploy1002: deploy-promote aborted:  (duration: 03m 22s)
* 16:44 brennen@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]]
* 16:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P26573 and previous config saved to /var/cache/conftool/dbconfig/20220426-164022-ladsgroup.json
* 16:36 klausman@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ml-staging-ctrl2002.codfw.wmnet
* 16:35 razzi@cumin1001: START - Cookbook sre.hosts.reimage for host clouddb1021.eqiad.wmnet with OS bullseye
* 16:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26572 and previous config saved to /var/cache/conftool/dbconfig/20220426-163535-ladsgroup.json
* 16:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:32 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786341{{!}}Set actor migration to read new for medium wikis (T275246)]] (duration: 02m 01s)
* 16:30 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-staging-ctrl2002.codfw.wmnet
* 16:28 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1021.eqiad.wmnet with reason: Upgrade to bullseye
* 16:28 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb1021.eqiad.wmnet with reason: Upgrade to bullseye
* 16:27 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26571 and previous config saved to /var/cache/conftool/dbconfig/20220426-162517-ladsgroup.json
* 16:22 bd808: Toolhub upgrade to 18d94d and post-deploy data migrations complete
* 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1163 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26570 and previous config saved to /var/cache/conftool/dbconfig/20220426-162244-ladsgroup.json
* 16:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 16:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 16:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26569 and previous config saved to /var/cache/conftool/dbconfig/20220426-162236-ladsgroup.json
* 16:22 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 16:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26568 and previous config saved to /var/cache/conftool/dbconfig/20220426-162029-ladsgroup.json
* 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26567 and previous config saved to /var/cache/conftool/dbconfig/20220426-161816-ladsgroup.json
* 16:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26566 and previous config saved to /var/cache/conftool/dbconfig/20220426-161808-ladsgroup.json
* 16:16 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:13 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 16:12 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 16:11 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 16:11 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:10 dancy@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): testing (duration: 00m 17s)
* 16:09 dancy@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): testing
* 16:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P26564 and previous config saved to /var/cache/conftool/dbconfig/20220426-160731-ladsgroup.json
* 16:06 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 16:06 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 16:04 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 16:03 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26563 and previous config saved to /var/cache/conftool/dbconfig/20220426-160303-ladsgroup.json
* 16:01 bd808@deploy1002: helmfile [staging] START helmfile.d/services/toolhub: apply
* 16:00 dancy@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 02m 43s)
* 15:58 dancy@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
* 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P26562 and previous config saved to /var/cache/conftool/dbconfig/20220426-155226-ladsgroup.json
* 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26561 and previous config saved to /var/cache/conftool/dbconfig/20220426-154758-ladsgroup.json
* 15:42 cmooney@cumin1001: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1 - cmooney@cumin1001
* 15:40 cmooney@cumin1001: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.4.1 - cmooney@cumin1001
* 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26560 and previous config saved to /var/cache/conftool/dbconfig/20220426-153720-ladsgroup.json
* 15:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26559 and previous config saved to /var/cache/conftool/dbconfig/20220426-153449-ladsgroup.json
* 15:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
* 15:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
* 15:34 klausman: Restarting pybal on lvs2009 to pick up change 786319 (ML staging k8s service setup)
* 15:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 15:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 15:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26558 and previous config saved to /var/cache/conftool/dbconfig/20220426-153253-ladsgroup.json
* 15:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26557 and previous config saved to /var/cache/conftool/dbconfig/20220426-153039-ladsgroup.json
* 15:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 15:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 15:28 klausman@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=ml-staging-ctrl2002.codfw.wmnet
* 15:27 klausman@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=ml-staging-ctrl2001.codfw.wmnet
* 15:24 klausman@puppetmaster1001: conftool action : set/pooled=yes,weight=10; selector: name=ml-staging-ctrl2002
* 15:24 klausman@puppetmaster1001: conftool action : set/pooled=yes,weight=10; selector: name=ml-staging-ctrl2001
* 15:14 klausman: Restarting pybal on lvs2010 to pick up change 786319 (ML staging k8s service setup)
* 15:12 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1007.eqiad.wmnet
* 14:56 vgutierrez: upgrading trafficserver to 8.0.8-1wm6 on cp4032 - [[phab:T304835|T304835]]
* 14:52 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1007.eqiad.wmnet
* 14:49 vgutierrez: upgrading trafficserver to 8.0.8-1wm6 on cp4026 - [[phab:T304835|T304835]]
* 14:44 vgutierrez: upload trafficserver 8.0.8-1wm6 to apt.wm.o (buster) - [[phab:T304835|T304835]]
* 14:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:28 klausman@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:25 klausman@cumin1001: START - Cookbook sre.dns.netbox
* 14:24 urbanecm@deploy1002: Synchronized README: no op (duration: 02m 11s)
* 14:21 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/: REVERT: Failed backports (duration: 01m 40s)
* 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:12 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:11 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:09 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:09 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:07 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:06 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:04 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 14:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:56 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host druid1006.eqiad.wmnet
* 13:56 tgr@deploy1002: scap failed: RuntimeError Scap failed!: 8/9 canaries failed their endpoint checks(https://en.wikipedia.org).  WARNING: canaries have not been rolled back. (duration: 02m 37s)
* 13:56 tgr@deploy1002: Scap failed!: 8/9 canaries failed their endpoint checks(https://en.wikipedia.org).  WARNING: canaries have not been rolled back.
* 13:53 tgr@deploy1002: Started scap: (no justification provided)
* 13:45 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1006.eqiad.wmnet
* 13:37 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:36 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1011 as pc1 primary [[phab:T306892|T306892]] (duration: 01m 37s)
* 13:29 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 13:29 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 13:28 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 13:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1011.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:23 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1011.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:22 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Rebooting pc1011 [[phab:T306892|T306892]]
* 13:21 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Rebooting pc1011 [[phab:T306892|T306892]]
* 13:21 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc1 primary [[phab:T306892|T306892]] (duration: 01m 07s)
* 13:18 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:14 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1005.eqiad.wmnet
* 13:07 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1005.eqiad.wmnet
* 12:55 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1004.eqiad.wmnet
* 12:52 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26550 and previous config saved to /var/cache/conftool/dbconfig/20220426-125244-kormat.json
* 12:48 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1004.eqiad.wmnet
* 12:46 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org
* 12:44 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org
* 12:37 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26547 and previous config saved to /var/cache/conftool/dbconfig/20220426-123740-kormat.json
* 12:24 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1015.eqiad.wmnet
* 12:22 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26546 and previous config saved to /var/cache/conftool/dbconfig/20220426-122235-kormat.json
* 12:14 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1015.eqiad.wmnet
* 12:07 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26545 and previous config saved to /var/cache/conftool/dbconfig/20220426-120731-kormat.json
* 12:07 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26544 and previous config saved to /var/cache/conftool/dbconfig/20220426-120727-kormat.json
* 12:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 12:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 12:00 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup1001.eqiad.wmnet with OS bullseye
* 11:52 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26543 and previous config saved to /var/cache/conftool/dbconfig/20220426-115223-kormat.json
* 11:49 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup1001.eqiad.wmnet with reason: host reimage
* 11:46 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup1001.eqiad.wmnet with reason: host reimage
* 11:42 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup2001.codfw.wmnet with OS bullseye
* 11:37 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26542 and previous config saved to /var/cache/conftool/dbconfig/20220426-113719-kormat.json
* 11:34 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host ms-backup1001.eqiad.wmnet with OS bullseye
* 11:30 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup2001.codfw.wmnet with reason: host reimage
* 11:27 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup2001.codfw.wmnet with reason: host reimage
* 11:23 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti-test2001.codfw.wmnet
* 11:22 topranks: Reconfigre routing policy lsw1-f1-eqiad, rename policies to use lower-case
* 11:22 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26541 and previous config saved to /var/cache/conftool/dbconfig/20220426-112215-kormat.json
* 11:17 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26540 and previous config saved to /var/cache/conftool/dbconfig/20220426-111751-kormat.json
* 11:17 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26539 and previous config saved to /var/cache/conftool/dbconfig/20220426-111741-kormat.json
* 11:17 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1170.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 11:17 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1170.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 11:16 topranks: Reconfigre routing policy lsw1-e1-eqiad, rename policies to use lower-case
* 11:13 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host ms-backup2001.codfw.wmnet with OS bullseye
* 11:11 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup1002.eqiad.wmnet with OS bullseye
* 11:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 11:09 topranks: Reconfigre routing policy lsw1-e2-eqiad, rename policies to use lower-case
* 11:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1014.eqiad.wmnet
* 11:08 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26538 and previous config saved to /var/cache/conftool/dbconfig/20220426-110819-kormat.json
* 11:05 topranks: Reconfigre routing policy lsw1-f2-eqiad, rename policies to use lower-case
* 11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1014.eqiad.wmnet
* 11:00 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup1002.eqiad.wmnet with reason: host reimage
* 10:57 topranks: Reconfigre routing policy lsw1-e3-eqiad, rename policies to use lower-case
* 10:57 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup1002.eqiad.wmnet with reason: host reimage
* 10:53 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 10:53 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26537 and previous config saved to /var/cache/conftool/dbconfig/20220426-105315-kormat.json
* 10:45 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host ms-backup1002.eqiad.wmnet with OS bullseye
* 10:44 topranks: Reconfigre routing policy lsw1-f3-eqiad, rename policies to use lower-case
* 10:43 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup2002.codfw.wmnet with OS bullseye
* 10:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2001.codfw.wmnet with OS bullseye
* 10:38 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26536 and previous config saved to /var/cache/conftool/dbconfig/20220426-103811-kormat.json
* 10:33 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 10:32 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup2002.codfw.wmnet with reason: host reimage
* 10:28 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup2002.codfw.wmnet with reason: host reimage
* 10:25 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1013.eqiad.wmnet
* 10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26535 and previous config saved to /var/cache/conftool/dbconfig/20220426-102307-kormat.json
* 10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26534 and previous config saved to /var/cache/conftool/dbconfig/20220426-102303-kormat.json
* 10:15 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1013.eqiad.wmnet
* 10:14 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host ms-backup2002.codfw.wmnet with OS bullseye
* 10:08 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26533 and previous config saved to /var/cache/conftool/dbconfig/20220426-100758-kormat.json
* 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2001.codfw.wmnet with reason: host reimage
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1122 into API', diff saved to https://phabricator.wikimedia.org/P26532 and previous config saved to /var/cache/conftool/dbconfig/20220426-100031-marostegui.json
* 10:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2001.codfw.wmnet with reason: host reimage
* 09:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P26531 and previous config saved to /var/cache/conftool/dbconfig/20220426-095957-root.json
* 09:56 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26530 and previous config saved to /var/cache/conftool/dbconfig/20220426-095627-kormat.json
* 09:52 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26528 and previous config saved to /var/cache/conftool/dbconfig/20220426-095254-kormat.json
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1002.eqiad.wmnet
* 09:51 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@9dbd5bc]: (no justification provided) (duration: 00m 07s)
* 09:51 nokafor@deploy1002: Started deploy [airflow-dags/analytics@9dbd5bc]: (no justification provided)
* 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1002.eqiad.wmnet
* 09:47 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1012.eqiad.wmnet
* 09:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2001.codfw.wmnet with OS bullseye
* 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P26526 and previous config saved to /var/cache/conftool/dbconfig/20220426-094453-root.json
* 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1001.eqiad.wmnet
* 09:41 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26525 and previous config saved to /var/cache/conftool/dbconfig/20220426-094123-kormat.json
* 09:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1001.eqiad.wmnet
* 09:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 09:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 09:37 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26524 and previous config saved to /var/cache/conftool/dbconfig/20220426-093750-kormat.json
* 09:36 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1012.eqiad.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf2002.codfw.wmnet
* 09:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 09:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 09:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 09:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 09:33 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26523 and previous config saved to /var/cache/conftool/dbconfig/20220426-093314-kormat.json
* 09:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:32 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:32 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 09:32 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 09:31 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf2002.codfw.wmnet
* 09:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf2001.codfw.wmnet
* 09:29 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 09:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P26522 and previous config saved to /var/cache/conftool/dbconfig/20220426-092949-root.json
* 09:29 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 09:27 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf2001.codfw.wmnet
* 09:26 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 09:26 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26521 and previous config saved to /var/cache/conftool/dbconfig/20220426-092619-kormat.json
* 09:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1115.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1115.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2093.codfw.wmnet,dborch1001.wikimedia.org with reason: Rebooting db1115 [[phab:T303174|T303174]]
* 09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2093.codfw.wmnet,dborch1001.wikimedia.org with reason: Rebooting db1115 [[phab:T303174|T303174]]
* 09:23 topranks: Reconfigure CR routers following bgp policy changes (no-op) CR785284
* 09:23 mvernon@cumin1001: conftool action : set/pooled=yes; selector: service=swift-fe,name=ms-fe1012.eqiad.wmnet
* 09:23 mvernon@cumin1001: conftool action : set/pooled=yes; selector: service=nginx,name=ms-fe1012.eqiad.wmnet
* 09:20 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1012.eqiad.wmnet
* 09:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P26520 and previous config saved to /var/cache/conftool/dbconfig/20220426-091445-root.json
* 09:14 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-fe1012.eqiad.wmnet
* 09:11 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26519 and previous config saved to /var/cache/conftool/dbconfig/20220426-091115-kormat.json
* 09:11 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26518 and previous config saved to /var/cache/conftool/dbconfig/20220426-091111-kormat.json
* 09:10 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26517 and previous config saved to /var/cache/conftool/dbconfig/20220426-091015-kormat.json
* 09:10 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:10 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P26516 and previous config saved to /var/cache/conftool/dbconfig/20220426-085941-root.json
* 08:56 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26515 and previous config saved to /var/cache/conftool/dbconfig/20220426-085607-kormat.json
* 08:47 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 08:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 1%: After reimage', diff saved to https://phabricator.wikimedia.org/P26514 and previous config saved to /var/cache/conftool/dbconfig/20220426-084437-root.json
* 08:43 jelto: pool name=mw229[7-9].codfw.wmnet, manual icinga recheck green after reboot
* 08:43 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw229[7-9].codfw.wmnet
* 08:41 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26513 and previous config saved to /var/cache/conftool/dbconfig/20220426-084103-kormat.json
* 08:34 moritzm: installing testvm2004 [[phab:T306499|T306499]]
* 08:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1122.eqiad.wmnet with OS bullseye
* 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1002.eqiad.wmnet
* 08:31 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 08:26 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26512 and previous config saved to /var/cache/conftool/dbconfig/20220426-082559-kormat.json
* 08:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host deploy1002.eqiad.wmnet
* 08:22 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26511 and previous config saved to /var/cache/conftool/dbconfig/20220426-082210-kormat.json
* 08:21 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26510 and previous config saved to /var/cache/conftool/dbconfig/20220426-082155-kormat.json
* 08:21 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1113.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:21 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1113.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1122.eqiad.wmnet with reason: host reimage
* 08:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1122.eqiad.wmnet with reason: host reimage
* 08:08 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1122.eqiad.wmnet with OS bullseye
* 08:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2004.codfw.wmnet
* 08:03 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2289.codfw.wmnet
* 07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2288.codfw.wmnet
* 07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2287.codfw.wmnet
* 07:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:48 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 07:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 07:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3005.wikimedia.org
* 07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 ([[phab:T302185|T302185]])', diff saved to https://phabricator.wikimedia.org/P26509 and previous config saved to /var/cache/conftool/dbconfig/20220426-073627-ladsgroup.json
* 07:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3005.wikimedia.org
* 07:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P26508 and previous config saved to /var/cache/conftool/dbconfig/20220426-072122-ladsgroup.json
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P26507 and previous config saved to /var/cache/conftool/dbconfig/20220426-070617-ladsgroup.json
* 07:01 marostegui: dbmaint s2@eqiad [[phab:T298554|T298554]]
* 06:54 jayme@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 03m 05s)
* 06:51 jayme@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 ([[phab:T302185|T302185]])', diff saved to https://phabricator.wikimedia.org/P26506 and previous config saved to /var/cache/conftool/dbconfig/20220426-065112-ladsgroup.json
* 06:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1109.eqiad.wmnet with OS bullseye
* 06:45 jayme: imported scap 4.7.0 to stretch-/buster-/bullseye-wikimedia - [[phab:T306827|T306827]]
* 06:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1109.eqiad.wmnet with reason: host reimage
* 06:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1109.eqiad.wmnet with reason: host reimage
* 06:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1109.eqiad.wmnet with OS bullseye
* 06:16 marostegui: dbmaint s2@eqiad [[phab:T300381|T300381]]
* 06:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1109 ([[phab:T302185|T302185]])', diff saved to https://phabricator.wikimedia.org/P26505 and previous config saved to /var/cache/conftool/dbconfig/20220426-061519-ladsgroup.json
* 06:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1109.eqiad.wmnet with reason: Maintenance
* 06:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1109.eqiad.wmnet with reason: Maintenance
* 06:14 marostegui: dbmaint s2@eqiad [[phab:T298557|T298557]]
* 06:07 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1100, s5 master from API', diff saved to https://phabricator.wikimedia.org/P26504 and previous config saved to /var/cache/conftool/dbconfig/20220426-060734-marostegui.json
* 06:06 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 is current s2 master, should not be in API [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26503 and previous config saved to /var/cache/conftool/dbconfig/20220426-060602-marostegui.json
* 06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1122 [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26502 and previous config saved to /var/cache/conftool/dbconfig/20220426-060344-root.json
* 06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26501 and previous config saved to /var/cache/conftool/dbconfig/20220426-060058-marostegui.json
* 06:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26500 and previous config saved to /var/cache/conftool/dbconfig/20220426-060033-marostegui.json
* 06:00 marostegui: Starting s2 eqiad failover from db1122 to db1162 - [[phab:T306417|T306417]]
* 04:54 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26498 and previous config saved to /var/cache/conftool/dbconfig/20220426-045406-root.json
* 04:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s2 [[phab:T306417|T306417]]
* 04:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s2 [[phab:T306417|T306417]]
* 04:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 04:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 04:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 04:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
== 2022-04-25 ==
* 23:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 23:01 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785948{{!}}ActorMigration: Read from rev_actor field in all of small wikis (T275246)]] (duration: 00m 57s)
* 22:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:49 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:612348{{!}}TimedMediaHandler: Make videojs the only player on all group1 (T248418)]] (duration: 00m 54s)
* 22:04 dancy@deploy1002: Synchronized README: testing scap mods (duration: 00m 54s)
* 22:00 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudweb2001-dev.wikimedia.org
* 21:56 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:52 eileen: civicrm revision {{Gerrit|7de7ddd4}} -> {{Gerrit|a841cf55}}
* 21:49 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 21:43 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudweb2001-dev.wikimedia.org
* 21:40 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
* 21:38 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:35 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 21:26 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
* 21:25 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785919{{!}}[Web scroll] Restore original sampling rate (T305442)]] (duration: 01m 01s)
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:59 dzahn@cumin2002: conftool action : set/pooled=inactive; selector: dc=codfw,name=mw2286.codfw.wmnet
* 20:58 mutante: rebooting mw2415
* 20:27 catrope@deploy1002: Synchronized wmf-config/wikitech.php: Config: [[gerrit:785399{{!}}labtestwiki: update labtest ldap server (T304881)]] (duration: 01m 39s)
* 20:23 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785908{{!}}Enable TOC for all users opted into modern Vector outside of pilot wikis (T306608)]] (duration: 01m 40s)
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: fresh role user
* 19:57 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: fresh role user
* 19:53 mutante: rebooting mw2414 through mw2419
* 19:46 dancy@deploy1002: Finished scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 12m 54s)
* 19:43 mutante: rebooting mw2412, mw2413
* 19:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
* 19:34 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
* 19:33 dancy@deploy1002: Started scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]]
* 19:30 dancy@deploy1002: Started scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]]
* 19:29 dancy@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 01m 43s)
* 19:27 dancy@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 01m 42s)
* 19:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:09 mutante: turning mw2412 through mw2419 into actual appservers - applying roles for the first time, will cause alerts probably
* 19:09 cwhite: install grafana-plugins 0.5 and restart grafana on grafana1002 [[phab:T304583|T304583]]
* 18:47 cstone: payments-wiki revision changed from {{Gerrit|a3c69385}} to {{Gerrit|786dc94f}}
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26497 and previous config saved to /var/cache/conftool/dbconfig/20220425-175957-ladsgroup.json
* 17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26496 and previous config saved to /var/cache/conftool/dbconfig/20220425-174451-ladsgroup.json
* 17:39 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudservices[2002-2003]-dev.wikimedia.org
* 17:34 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:30 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26495 and previous config saved to /var/cache/conftool/dbconfig/20220425-172946-ladsgroup.json
* 17:24 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudservices[2002-2003]-dev.wikimedia.org
* 17:17 aokoth@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM otrs1001.eqiad.wmnet
* 17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26494 and previous config saved to /var/cache/conftool/dbconfig/20220425-171441-ladsgroup.json
* 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26493 and previous config saved to /var/cache/conftool/dbconfig/20220425-171223-ladsgroup.json
* 17:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 17:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 17:04 aokoth@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM otrs1001.eqiad.wmnet
* 16:47 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-eqiad cluster: Reboot kafka nodes
* 15:56 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
* 15:56 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
* 15:54 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1001.eqiad.wmnet
* 15:47 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki1001.eqiad.wmnet
* 15:46 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
* 15:46 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
* 15:41 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:37 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-eqiad cluster: Reboot kafka nodes
* 15:36 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 15:32 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-codfw cluster: Reboot kafka nodes
* 15:25 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 15:23 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:21 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 15:20 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1001.eqiad.wmnet
* 15:18 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2001.codfw.wmnet
* 15:16 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki-root1001.eqiad.wmnet
* 15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 15:15 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
* 15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 15:15 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 15:13 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
* 15:12 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 15:10 jbond@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "test sync - jbond@cumin1001"
* 15:09 jbond@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test sync - jbond@cumin1001"
* 14:52 krinkle@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|I22240af06d}} (duration: 01m 42s)
* 14:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:21 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-codfw cluster: Reboot kafka nodes
* 14:13 jelto: mw2253: remove puppet lock of stuck puppet run due to reboot, run-puppet-agent
* 14:10 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1011.eqiad.wmnet
* 14:00 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1011.eqiad.wmnet
* 13:51 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1010.eqiad.wmnet
* 13:41 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1010.eqiad.wmnet
* 13:39 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1003.eqiad.wmnet
* 13:35 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1003.eqiad.wmnet
* 13:31 jelto: maintenance (rolling reboot) on api_appserver in codfw (cookbook sre.hosts.reboot-cluster -D codfw -c api_appserver --percentage 5 --grace_sleep 60)
* 13:30 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 13:28 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1002.eqiad.wmnet
* 13:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1002.eqiad.wmnet
* 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26492 and previous config saved to /var/cache/conftool/dbconfig/20220425-131411-ladsgroup.json
* 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:13 urbanecm: UTC afternoon B&C window done
* 13:12 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/CentralAuth/includes/User/GlobalUserSelectQueryBuilder.php: {{Gerrit|c4c4c3219ad69705c83caa50754d95285c96f352}}: GlobalUserSelectQueryBuilder: Do not fatal when no users are returned ([[phab:T306535|T306535]]) (duration: 00m 54s)
* 13:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:04 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0338c9bb9ff6ad388270d1e2df5013d8a49d1210}}: GrowthExperiments: Do not use facebook in campaign pattern ([[phab:T303785|T303785]]) (duration: 00m 51s)
* 13:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2001.wikimedia.org
* 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26491 and previous config saved to /var/cache/conftool/dbconfig/20220425-125906-ladsgroup.json
* 12:58 krinkle@deploy1002: Synchronized private/PrivateSettings.php: {{Gerrit|If4d7ea24b4db59}} (duration: 00m 59s)
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2001.wikimedia.org
* 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26490 and previous config saved to /var/cache/conftool/dbconfig/20220425-124401-ladsgroup.json
* 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26489 and previous config saved to /var/cache/conftool/dbconfig/20220425-122856-ladsgroup.json
* 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26488 and previous config saved to /var/cache/conftool/dbconfig/20220425-122531-ladsgroup.json
* 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26487 and previous config saved to /var/cache/conftool/dbconfig/20220425-122518-ladsgroup.json
* 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26485 and previous config saved to /var/cache/conftool/dbconfig/20220425-121013-ladsgroup.json
* 12:02 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1001.eqiad.wmnet
* 11:58 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1001.eqiad.wmnet
* 11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26484 and previous config saved to /var/cache/conftool/dbconfig/20220425-115508-ladsgroup.json
* 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26483 and previous config saved to /var/cache/conftool/dbconfig/20220425-114003-ladsgroup.json
* 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26482 and previous config saved to /var/cache/conftool/dbconfig/20220425-113138-ladsgroup.json
* 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26481 and previous config saved to /var/cache/conftool/dbconfig/20220425-113130-ladsgroup.json
* 11:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
* 11:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 11:20 moritzm: failover Ganeti master in codfw-test to ganeti-test2003 [[phab:T306499|T306499]]
* 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26480 and previous config saved to /var/cache/conftool/dbconfig/20220425-111625-ladsgroup.json
* 11:13 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26479 and previous config saved to /var/cache/conftool/dbconfig/20220425-111315-kormat.json
* 11:11 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host an-master1001.eqiad.wmnet
* 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26478 and previous config saved to /var/cache/conftool/dbconfig/20220425-110119-ladsgroup.json
* 11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1001.eqiad.wmnet
* 10:58 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26477 and previous config saved to /var/cache/conftool/dbconfig/20220425-105811-kormat.json
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26476 and previous config saved to /var/cache/conftool/dbconfig/20220425-104614-ladsgroup.json
* 10:45 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:45 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:43 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26475 and previous config saved to /var/cache/conftool/dbconfig/20220425-104307-kormat.json
* 10:38 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:38 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:28 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26474 and previous config saved to /var/cache/conftool/dbconfig/20220425-102803-kormat.json
* 10:24 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:24 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flowspec1001.eqiad.wmnet
* 10:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host flowspec1001.eqiad.wmnet
* 10:14 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
* 10:13 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:07 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2419.codfw.wmnet
* 10:05 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:04 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:02 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2419.codfw.wmnet
* 10:02 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2418.codfw.wmnet
* 09:56 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2418.codfw.wmnet
* 09:56 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2417.codfw.wmnet
* 09:55 kormat@cumin1001: dbctl commit (dc=all): 'db1137 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26472 and previous config saved to /var/cache/conftool/dbconfig/20220425-095543-kormat.json
* 09:55 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:55 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:51 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2417.codfw.wmnet
* 09:50 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2416.codfw.wmnet
* 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26471 and previous config saved to /var/cache/conftool/dbconfig/20220425-094600-ladsgroup.json
* 09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 09:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26470 and previous config saved to /var/cache/conftool/dbconfig/20220425-094552-ladsgroup.json
* 09:43 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2416.codfw.wmnet
* 09:42 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2415.codfw.wmnet
* 09:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 09:41 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 09:37 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2415.codfw.wmnet
* 09:36 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2414.codfw.wmnet
* 09:31 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2414.codfw.wmnet
* 09:30 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2413.codfw.wmnet
* 09:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26469 and previous config saved to /var/cache/conftool/dbconfig/20220425-093047-ladsgroup.json
* 09:28 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26468 and previous config saved to /var/cache/conftool/dbconfig/20220425-092807-kormat.json
* 09:25 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2413.codfw.wmnet
* 09:23 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2412.codfw.wmnet
* 09:16 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2412.codfw.wmnet
* 09:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26467 and previous config saved to /var/cache/conftool/dbconfig/20220425-091542-ladsgroup.json
* 09:13 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26466 and previous config saved to /var/cache/conftool/dbconfig/20220425-091303-kormat.json
* 09:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26465 and previous config saved to /var/cache/conftool/dbconfig/20220425-090037-ladsgroup.json
* 08:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1002.eqiad.wmnet
* 08:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26464 and previous config saved to /var/cache/conftool/dbconfig/20220425-085822-ladsgroup.json
* 08:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 08:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet
* 08:58 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26463 and previous config saved to /var/cache/conftool/dbconfig/20220425-085759-kormat.json
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26462 and previous config saved to /var/cache/conftool/dbconfig/20220425-085650-ladsgroup.json
* 08:55 vgutierrez: restart varnish and ats on cp2037 to clear daemon restart alerts
* 08:54 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1002.eqiad.wmnet
* 08:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet
* 08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 9 hosts with reason: Rebooting db1154 [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 9 hosts with reason: Rebooting db1154 [[phab:T303174|T303174]]
* 08:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26461 and previous config saved to /var/cache/conftool/dbconfig/20220425-084318-ladsgroup.json
* 08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26460 and previous config saved to /var/cache/conftool/dbconfig/20220425-084256-kormat.json
* 08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26459 and previous config saved to /var/cache/conftool/dbconfig/20220425-084251-kormat.json
* 08:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26458 and previous config saved to /var/cache/conftool/dbconfig/20220425-084145-ladsgroup.json
* 08:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26457 and previous config saved to /var/cache/conftool/dbconfig/20220425-082813-ladsgroup.json
* 08:27 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26456 and previous config saved to /var/cache/conftool/dbconfig/20220425-082747-kormat.json
* 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26455 and previous config saved to /var/cache/conftool/dbconfig/20220425-082640-ladsgroup.json
* 08:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26454 and previous config saved to /var/cache/conftool/dbconfig/20220425-081307-ladsgroup.json
* 08:12 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26453 and previous config saved to /var/cache/conftool/dbconfig/20220425-081244-kormat.json
* 08:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26452 and previous config saved to /var/cache/conftool/dbconfig/20220425-081135-ladsgroup.json
* 08:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26451 and previous config saved to /var/cache/conftool/dbconfig/20220425-080910-ladsgroup.json
* 08:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 08:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26450 and previous config saved to /var/cache/conftool/dbconfig/20220425-080838-ladsgroup.json
* 07:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader2001.codfw.wmnet
* 07:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26449 and previous config saved to /var/cache/conftool/dbconfig/20220425-075801-ladsgroup.json
* 07:57 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26448 and previous config saved to /var/cache/conftool/dbconfig/20220425-075740-kormat.json
* 07:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26447 and previous config saved to /var/cache/conftool/dbconfig/20220425-075333-ladsgroup.json
* 07:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader2001.codfw.wmnet
* 07:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader1001.eqiad.wmnet
* 07:51 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26446 and previous config saved to /var/cache/conftool/dbconfig/20220425-075106-kormat.json
* 07:50 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26445 and previous config saved to /var/cache/conftool/dbconfig/20220425-075045-kormat.json
* 07:50 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 07:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 07:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org
* 07:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader1001.eqiad.wmnet
* 07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26444 and previous config saved to /var/cache/conftool/dbconfig/20220425-074912-ladsgroup.json
* 07:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 07:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26443 and previous config saved to /var/cache/conftool/dbconfig/20220425-074904-ladsgroup.json
* 07:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org
* 07:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26442 and previous config saved to /var/cache/conftool/dbconfig/20220425-073828-ladsgroup.json
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26441 and previous config saved to /var/cache/conftool/dbconfig/20220425-073359-ladsgroup.json
* 07:31 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host seaborgium.wikimedia.org
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26440 and previous config saved to /var/cache/conftool/dbconfig/20220425-072323-ladsgroup.json
* 07:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org
* 07:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26439 and previous config saved to /var/cache/conftool/dbconfig/20220425-072157-ladsgroup.json
* 07:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 07:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26438 and previous config saved to /var/cache/conftool/dbconfig/20220425-072149-ladsgroup.json
* 07:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2005.codfw.wmnet
* 07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26437 and previous config saved to /var/cache/conftool/dbconfig/20220425-071853-ladsgroup.json
* 07:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2005.codfw.wmnet
* 07:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2004.codfw.wmnet
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:11 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784685{{!}}ActorMigration: Start reading from rev_actor field in group0 (T275246)]] (duration: 00m 50s)
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2004.codfw.wmnet
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26436 and previous config saved to /var/cache/conftool/dbconfig/20220425-070644-ladsgroup.json
* 07:06 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785326{{!}}Add tothemoon.ser.asu.edu to the wgCopyUploadsDomains allowlist of commonswiki (T306671)]] (duration: 00m 52s)
* 07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26435 and previous config saved to /var/cache/conftool/dbconfig/20220425-070348-ladsgroup.json
* 06:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1001.wikimedia.org
* 06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26434 and previous config saved to /var/cache/conftool/dbconfig/20220425-065559-ladsgroup.json
* 06:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 06:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 06:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1001.wikimedia.org
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26433 and previous config saved to /var/cache/conftool/dbconfig/20220425-065139-ladsgroup.json
* 06:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1132 into s1 [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P26432 and previous config saved to /var/cache/conftool/dbconfig/20220425-063823-marostegui.json
* 06:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26431 and previous config saved to /var/cache/conftool/dbconfig/20220425-063634-ladsgroup.json
* 06:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26430 and previous config saved to /var/cache/conftool/dbconfig/20220425-063409-ladsgroup.json
* 06:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:11 dcausse: depooling and restarting blazegraph on wdqs1007 (deadlocked for 4+days)
* 04:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26429 and previous config saved to /var/cache/conftool/dbconfig/20220425-045902-ladsgroup.json
* 04:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26428 and previous config saved to /var/cache/conftool/dbconfig/20220425-044357-ladsgroup.json
* 04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26427 and previous config saved to /var/cache/conftool/dbconfig/20220425-042852-ladsgroup.json
* 04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26426 and previous config saved to /var/cache/conftool/dbconfig/20220425-041347-ladsgroup.json
* 04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26425 and previous config saved to /var/cache/conftool/dbconfig/20220425-040940-ladsgroup.json
* 04:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 04:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26424 and previous config saved to /var/cache/conftool/dbconfig/20220425-040926-ladsgroup.json
* 03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26423 and previous config saved to /var/cache/conftool/dbconfig/20220425-035421-ladsgroup.json
* 03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26422 and previous config saved to /var/cache/conftool/dbconfig/20220425-033916-ladsgroup.json
* 03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26421 and previous config saved to /var/cache/conftool/dbconfig/20220425-032410-ladsgroup.json
* 03:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26420 and previous config saved to /var/cache/conftool/dbconfig/20220425-031959-ladsgroup.json
* 03:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26419 and previous config saved to /var/cache/conftool/dbconfig/20220425-031944-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26418 and previous config saved to /var/cache/conftool/dbconfig/20220425-030439-ladsgroup.json
* 02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26417 and previous config saved to /var/cache/conftool/dbconfig/20220425-024934-ladsgroup.json
* 02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26416 and previous config saved to /var/cache/conftool/dbconfig/20220425-023429-ladsgroup.json
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26415 and previous config saved to /var/cache/conftool/dbconfig/20220425-023020-ladsgroup.json
* 02:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26414 and previous config saved to /var/cache/conftool/dbconfig/20220425-023012-ladsgroup.json
* 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26413 and previous config saved to /var/cache/conftool/dbconfig/20220425-021507-ladsgroup.json
* 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26412 and previous config saved to /var/cache/conftool/dbconfig/20220425-020002-ladsgroup.json
* 01:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26411 and previous config saved to /var/cache/conftool/dbconfig/20220425-014457-ladsgroup.json
* 01:40 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: update new codfw1dev host (duration: 00m 54s)
* 01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
* 01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26410 and previous config saved to /var/cache/conftool/dbconfig/20220425-010952-ladsgroup.json
* 01:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26409 and previous config saved to /var/cache/conftool/dbconfig/20220425-010938-ladsgroup.json
* 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26408 and previous config saved to /var/cache/conftool/dbconfig/20220425-005432-ladsgroup.json
* 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26407 and previous config saved to /var/cache/conftool/dbconfig/20220425-003927-ladsgroup.json
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26406 and previous config saved to /var/cache/conftool/dbconfig/20220425-002422-ladsgroup.json
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26405 and previous config saved to /var/cache/conftool/dbconfig/20220425-001152-ladsgroup.json
* 00:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 00:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26404 and previous config saved to /var/cache/conftool/dbconfig/20220425-001144-ladsgroup.json
== 2022-04-24 ==
* 23:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26403 and previous config saved to /var/cache/conftool/dbconfig/20220424-235639-ladsgroup.json
* 23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26402 and previous config saved to /var/cache/conftool/dbconfig/20220424-234134-ladsgroup.json
* 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26401 and previous config saved to /var/cache/conftool/dbconfig/20220424-232629-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26400 and previous config saved to /var/cache/conftool/dbconfig/20220424-232219-ladsgroup.json
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26399 and previous config saved to /var/cache/conftool/dbconfig/20220424-232205-ladsgroup.json
* 23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26398 and previous config saved to /var/cache/conftool/dbconfig/20220424-230700-ladsgroup.json
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26397 and previous config saved to /var/cache/conftool/dbconfig/20220424-230136-ladsgroup.json
* 22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26396 and previous config saved to /var/cache/conftool/dbconfig/20220424-225155-ladsgroup.json
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26395 and previous config saved to /var/cache/conftool/dbconfig/20220424-224631-ladsgroup.json
* 22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26394 and previous config saved to /var/cache/conftool/dbconfig/20220424-223650-ladsgroup.json
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26393 and previous config saved to /var/cache/conftool/dbconfig/20220424-223240-ladsgroup.json
* 22:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26392 and previous config saved to /var/cache/conftool/dbconfig/20220424-223232-ladsgroup.json
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26391 and previous config saved to /var/cache/conftool/dbconfig/20220424-223126-ladsgroup.json
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26390 and previous config saved to /var/cache/conftool/dbconfig/20220424-221727-ladsgroup.json
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26389 and previous config saved to /var/cache/conftool/dbconfig/20220424-221621-ladsgroup.json
* 22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26388 and previous config saved to /var/cache/conftool/dbconfig/20220424-220222-ladsgroup.json
* 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26387 and previous config saved to /var/cache/conftool/dbconfig/20220424-214717-ladsgroup.json
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26386 and previous config saved to /var/cache/conftool/dbconfig/20220424-213440-ladsgroup.json
* 21:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26385 and previous config saved to /var/cache/conftool/dbconfig/20220424-213425-ladsgroup.json
* 21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26384 and previous config saved to /var/cache/conftool/dbconfig/20220424-211920-ladsgroup.json
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26383 and previous config saved to /var/cache/conftool/dbconfig/20220424-211607-ladsgroup.json
* 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26382 and previous config saved to /var/cache/conftool/dbconfig/20220424-211559-ladsgroup.json
* 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26381 and previous config saved to /var/cache/conftool/dbconfig/20220424-210521-ladsgroup.json
* 21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26380 and previous config saved to /var/cache/conftool/dbconfig/20220424-210415-ladsgroup.json
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26379 and previous config saved to /var/cache/conftool/dbconfig/20220424-210052-ladsgroup.json
* 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26378 and previous config saved to /var/cache/conftool/dbconfig/20220424-205016-ladsgroup.json
* 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26377 and previous config saved to /var/cache/conftool/dbconfig/20220424-204910-ladsgroup.json
* 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26376 and previous config saved to /var/cache/conftool/dbconfig/20220424-204547-ladsgroup.json
* 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26375 and previous config saved to /var/cache/conftool/dbconfig/20220424-203639-ladsgroup.json
* 20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26374 and previous config saved to /var/cache/conftool/dbconfig/20220424-203630-ladsgroup.json
* 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26373 and previous config saved to /var/cache/conftool/dbconfig/20220424-203511-ladsgroup.json
* 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26372 and previous config saved to /var/cache/conftool/dbconfig/20220424-203042-ladsgroup.json
* 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26371 and previous config saved to /var/cache/conftool/dbconfig/20220424-202125-ladsgroup.json
* 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26370 and previous config saved to /var/cache/conftool/dbconfig/20220424-202006-ladsgroup.json
* 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26369 and previous config saved to /var/cache/conftool/dbconfig/20220424-200620-ladsgroup.json
* 19:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26368 and previous config saved to /var/cache/conftool/dbconfig/20220424-195115-ladsgroup.json
* 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26367 and previous config saved to /var/cache/conftool/dbconfig/20220424-194705-ladsgroup.json
* 19:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26366 and previous config saved to /var/cache/conftool/dbconfig/20220424-194651-ladsgroup.json
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26365 and previous config saved to /var/cache/conftool/dbconfig/20220424-193635-ladsgroup.json
* 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26364 and previous config saved to /var/cache/conftool/dbconfig/20220424-193611-ladsgroup.json
* 19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26363 and previous config saved to /var/cache/conftool/dbconfig/20220424-193146-ladsgroup.json
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26362 and previous config saved to /var/cache/conftool/dbconfig/20220424-193028-ladsgroup.json
* 19:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 19:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26361 and previous config saved to /var/cache/conftool/dbconfig/20220424-193020-ladsgroup.json
* 19:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26360 and previous config saved to /var/cache/conftool/dbconfig/20220424-192106-ladsgroup.json
* 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26359 and previous config saved to /var/cache/conftool/dbconfig/20220424-191641-ladsgroup.json
* 19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26358 and previous config saved to /var/cache/conftool/dbconfig/20220424-191515-ladsgroup.json
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26357 and previous config saved to /var/cache/conftool/dbconfig/20220424-190601-ladsgroup.json
* 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26356 and previous config saved to /var/cache/conftool/dbconfig/20220424-190135-ladsgroup.json
* 19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26355 and previous config saved to /var/cache/conftool/dbconfig/20220424-190008-ladsgroup.json
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26354 and previous config saved to /var/cache/conftool/dbconfig/20220424-185724-ladsgroup.json
* 18:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26353 and previous config saved to /var/cache/conftool/dbconfig/20220424-185717-ladsgroup.json
* 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26352 and previous config saved to /var/cache/conftool/dbconfig/20220424-185056-ladsgroup.json
* 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26351 and previous config saved to /var/cache/conftool/dbconfig/20220424-184503-ladsgroup.json
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26350 and previous config saved to /var/cache/conftool/dbconfig/20220424-184212-ladsgroup.json
* 18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26349 and previous config saved to /var/cache/conftool/dbconfig/20220424-183813-ladsgroup.json
* 18:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26348 and previous config saved to /var/cache/conftool/dbconfig/20220424-183805-ladsgroup.json
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26347 and previous config saved to /var/cache/conftool/dbconfig/20220424-182707-ladsgroup.json
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26346 and previous config saved to /var/cache/conftool/dbconfig/20220424-182300-ladsgroup.json
* 18:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26345 and previous config saved to /var/cache/conftool/dbconfig/20220424-181201-ladsgroup.json
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26344 and previous config saved to /var/cache/conftool/dbconfig/20220424-180755-ladsgroup.json
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26343 and previous config saved to /var/cache/conftool/dbconfig/20220424-180555-ladsgroup.json
* 18:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26342 and previous config saved to /var/cache/conftool/dbconfig/20220424-180530-ladsgroup.json
* 18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26341 and previous config saved to /var/cache/conftool/dbconfig/20220424-180013-ladsgroup.json
* 18:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26340 and previous config saved to /var/cache/conftool/dbconfig/20220424-175250-ladsgroup.json
* 17:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26339 and previous config saved to /var/cache/conftool/dbconfig/20220424-175025-ladsgroup.json
* 17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26338 and previous config saved to /var/cache/conftool/dbconfig/20220424-174553-ladsgroup.json
* 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26337 and previous config saved to /var/cache/conftool/dbconfig/20220424-173955-ladsgroup.json
* 17:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26336 and previous config saved to /var/cache/conftool/dbconfig/20220424-173520-ladsgroup.json
* 17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26335 and previous config saved to /var/cache/conftool/dbconfig/20220424-172450-ladsgroup.json
* 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26334 and previous config saved to /var/cache/conftool/dbconfig/20220424-172015-ladsgroup.json
* 17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26333 and previous config saved to /var/cache/conftool/dbconfig/20220424-170945-ladsgroup.json
* 16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26332 and previous config saved to /var/cache/conftool/dbconfig/20220424-165439-ladsgroup.json
* 16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26331 and previous config saved to /var/cache/conftool/dbconfig/20220424-164748-ladsgroup.json
* 16:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 16:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26330 and previous config saved to /var/cache/conftool/dbconfig/20220424-163151-ladsgroup.json
* 16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 16:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 03:20 Amir1: optimizing flaggedtemplates on plwiki (s2) in db2088
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26329 and previous config saved to /var/cache/conftool/dbconfig/20220424-012830-ladsgroup.json
* 01:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26328 and previous config saved to /var/cache/conftool/dbconfig/20220424-011325-ladsgroup.json
* 00:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26327 and previous config saved to /var/cache/conftool/dbconfig/20220424-005820-ladsgroup.json
* 00:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26326 and previous config saved to /var/cache/conftool/dbconfig/20220424-004315-ladsgroup.json
== 2022-04-23 ==
* 23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26325 and previous config saved to /var/cache/conftool/dbconfig/20220423-232748-ladsgroup.json
* 23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26324 and previous config saved to /var/cache/conftool/dbconfig/20220423-232735-ladsgroup.json
* 23:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P26323 and previous config saved to /var/cache/conftool/dbconfig/20220423-231230-ladsgroup.json
* 22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P26322 and previous config saved to /var/cache/conftool/dbconfig/20220423-225725-ladsgroup.json
* 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26321 and previous config saved to /var/cache/conftool/dbconfig/20220423-224220-ladsgroup.json
* 21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26320 and previous config saved to /var/cache/conftool/dbconfig/20220423-212332-ladsgroup.json
* 21:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 21:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26319 and previous config saved to /var/cache/conftool/dbconfig/20220423-212324-ladsgroup.json
* 21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P26318 and previous config saved to /var/cache/conftool/dbconfig/20220423-210819-ladsgroup.json
* 20:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P26317 and previous config saved to /var/cache/conftool/dbconfig/20220423-205313-ladsgroup.json
* 20:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26316 and previous config saved to /var/cache/conftool/dbconfig/20220423-203808-ladsgroup.json
* 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26315 and previous config saved to /var/cache/conftool/dbconfig/20220423-191224-ladsgroup.json
* 19:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 19:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26314 and previous config saved to /var/cache/conftool/dbconfig/20220423-191216-ladsgroup.json
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P26313 and previous config saved to /var/cache/conftool/dbconfig/20220423-185711-ladsgroup.json
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P26312 and previous config saved to /var/cache/conftool/dbconfig/20220423-184206-ladsgroup.json
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26311 and previous config saved to /var/cache/conftool/dbconfig/20220423-182701-ladsgroup.json
* 16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26310 and previous config saved to /var/cache/conftool/dbconfig/20220423-165939-ladsgroup.json
* 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 16:25 akosiaris: increase mw1335 and mw1336 weights on the jobrunner cluster from 1 to 4 (they were at %25 CPU usage). That should direct more traffic to them and lighten the load on the rest.
* 16:24 akosiaris@cumin1001: conftool action : set/weight=4; selector: cluster=jobrunner,name=mw1336.eqiad.wmnet
* 16:24 akosiaris@cumin1001: conftool action : set/weight=4; selector: cluster=jobrunner,name=mw1335.eqiad.wmnet
* 16:22 akosiaris@cumin1001: conftool action : set/weight=8; selector: cluster=jobrunner,name=mw1335.eqiad.wmnet
* 16:22 akosiaris@cumin1001: conftool action : set/weight=8; selector: cluster=jobrunner,name=mw1336.eqiad.wmnet
* 16:18 akosiaris: depool the videoscalers from the jobrunner cluster. Effectively split the 2 clusters that way. This should isolate the rest of the jobs from the video transcoding jobs reducing the latency that they are experiencing
* 16:17 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1446.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1445.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1440.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1439.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1438.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1437.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1338.eqiad.wmnet
* 15:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 15:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26309 and previous config saved to /var/cache/conftool/dbconfig/20220423-142129-ladsgroup.json
* 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26308 and previous config saved to /var/cache/conftool/dbconfig/20220423-140624-ladsgroup.json
* 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26307 and previous config saved to /var/cache/conftool/dbconfig/20220423-135119-ladsgroup.json
* 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26306 and previous config saved to /var/cache/conftool/dbconfig/20220423-133614-ladsgroup.json
* 12:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26305 and previous config saved to /var/cache/conftool/dbconfig/20220423-123558-ladsgroup.json
* 12:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 12:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26304 and previous config saved to /var/cache/conftool/dbconfig/20220423-123550-ladsgroup.json
* 12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P26303 and previous config saved to /var/cache/conftool/dbconfig/20220423-122045-ladsgroup.json
* 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P26302 and previous config saved to /var/cache/conftool/dbconfig/20220423-120540-ladsgroup.json
* 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26301 and previous config saved to /var/cache/conftool/dbconfig/20220423-115035-ladsgroup.json
* 11:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26300 and previous config saved to /var/cache/conftool/dbconfig/20220423-110511-ladsgroup.json
* 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26299 and previous config saved to /var/cache/conftool/dbconfig/20220423-105005-ladsgroup.json
* 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26298 and previous config saved to /var/cache/conftool/dbconfig/20220423-103500-ladsgroup.json
* 10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26297 and previous config saved to /var/cache/conftool/dbconfig/20220423-103135-ladsgroup.json
* 10:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 10:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26296 and previous config saved to /var/cache/conftool/dbconfig/20220423-103127-ladsgroup.json
* 10:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26295 and previous config saved to /var/cache/conftool/dbconfig/20220423-101955-ladsgroup.json
* 10:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P26294 and previous config saved to /var/cache/conftool/dbconfig/20220423-101622-ladsgroup.json
* 10:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P26293 and previous config saved to /var/cache/conftool/dbconfig/20220423-100115-ladsgroup.json
* 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26292 and previous config saved to /var/cache/conftool/dbconfig/20220423-094610-ladsgroup.json
* 09:38 elukey: `apt-get clean` on an-airflow1001 to free some space
* 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26291 and previous config saved to /var/cache/conftool/dbconfig/20220423-093443-ladsgroup.json
* 09:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26290 and previous config saved to /var/cache/conftool/dbconfig/20220423-093435-ladsgroup.json
* 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26289 and previous config saved to /var/cache/conftool/dbconfig/20220423-091930-ladsgroup.json
* 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26288 and previous config saved to /var/cache/conftool/dbconfig/20220423-090425-ladsgroup.json
* 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26287 and previous config saved to /var/cache/conftool/dbconfig/20220423-084920-ladsgroup.json
* 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26286 and previous config saved to /var/cache/conftool/dbconfig/20220423-083545-ladsgroup.json
* 08:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26285 and previous config saved to /var/cache/conftool/dbconfig/20220423-083532-ladsgroup.json
* 08:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26284 and previous config saved to /var/cache/conftool/dbconfig/20220423-082735-ladsgroup.json
* 08:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 08:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 08:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26283 and previous config saved to /var/cache/conftool/dbconfig/20220423-082726-ladsgroup.json
* 08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26282 and previous config saved to /var/cache/conftool/dbconfig/20220423-082027-ladsgroup.json
* 08:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26281 and previous config saved to /var/cache/conftool/dbconfig/20220423-081221-ladsgroup.json
* 08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26280 and previous config saved to /var/cache/conftool/dbconfig/20220423-080522-ladsgroup.json
* 07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26279 and previous config saved to /var/cache/conftool/dbconfig/20220423-075716-ladsgroup.json
* 07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26278 and previous config saved to /var/cache/conftool/dbconfig/20220423-075017-ladsgroup.json
* 07:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26277 and previous config saved to /var/cache/conftool/dbconfig/20220423-074211-ladsgroup.json
* 07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26276 and previous config saved to /var/cache/conftool/dbconfig/20220423-073656-ladsgroup.json
* 07:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 07:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26275 and previous config saved to /var/cache/conftool/dbconfig/20220423-073648-ladsgroup.json
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26274 and previous config saved to /var/cache/conftool/dbconfig/20220423-072143-ladsgroup.json
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26273 and previous config saved to /var/cache/conftool/dbconfig/20220423-070638-ladsgroup.json
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26272 and previous config saved to /var/cache/conftool/dbconfig/20220423-065133-ladsgroup.json
* 06:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26271 and previous config saved to /var/cache/conftool/dbconfig/20220423-062503-ladsgroup.json
* 06:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 06:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 06:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26270 and previous config saved to /var/cache/conftool/dbconfig/20220423-062455-ladsgroup.json
* 06:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P26269 and previous config saved to /var/cache/conftool/dbconfig/20220423-060950-ladsgroup.json
* 05:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P26268 and previous config saved to /var/cache/conftool/dbconfig/20220423-055445-ladsgroup.json
* 05:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26267 and previous config saved to /var/cache/conftool/dbconfig/20220423-055118-ladsgroup.json
* 05:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 05:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26266 and previous config saved to /var/cache/conftool/dbconfig/20220423-053940-ladsgroup.json
* 05:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 05:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 05:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26265 and previous config saved to /var/cache/conftool/dbconfig/20220423-051219-ladsgroup.json
* 04:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26264 and previous config saved to /var/cache/conftool/dbconfig/20220423-045714-ladsgroup.json
* 04:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26263 and previous config saved to /var/cache/conftool/dbconfig/20220423-044209-ladsgroup.json
* 04:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26262 and previous config saved to /var/cache/conftool/dbconfig/20220423-042704-ladsgroup.json
* 04:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26261 and previous config saved to /var/cache/conftool/dbconfig/20220423-042001-ladsgroup.json
* 04:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 04:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 04:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26260 and previous config saved to /var/cache/conftool/dbconfig/20220423-041953-ladsgroup.json
* 04:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P26259 and previous config saved to /var/cache/conftool/dbconfig/20220423-040448-ladsgroup.json
* 03:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P26258 and previous config saved to /var/cache/conftool/dbconfig/20220423-034943-ladsgroup.json
* 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26257 and previous config saved to /var/cache/conftool/dbconfig/20220423-034558-ladsgroup.json
* 03:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 03:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 03:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26256 and previous config saved to /var/cache/conftool/dbconfig/20220423-034550-ladsgroup.json
* 03:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26255 and previous config saved to /var/cache/conftool/dbconfig/20220423-033438-ladsgroup.json
* 03:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26254 and previous config saved to /var/cache/conftool/dbconfig/20220423-033045-ladsgroup.json
* 03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26253 and previous config saved to /var/cache/conftool/dbconfig/20220423-031540-ladsgroup.json
* 03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26252 and previous config saved to /var/cache/conftool/dbconfig/20220423-030035-ladsgroup.json
* 02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26251 and previous config saved to /var/cache/conftool/dbconfig/20220423-021851-ladsgroup.json
* 02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26250 and previous config saved to /var/cache/conftool/dbconfig/20220423-021826-ladsgroup.json
* 02:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 02:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 02:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26249 and previous config saved to /var/cache/conftool/dbconfig/20220423-021211-ladsgroup.json
* 02:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 02:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 02:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26248 and previous config saved to /var/cache/conftool/dbconfig/20220423-020346-ladsgroup.json
* 01:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26247 and previous config saved to /var/cache/conftool/dbconfig/20220423-014841-ladsgroup.json
* 01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26246 and previous config saved to /var/cache/conftool/dbconfig/20220423-013450-ladsgroup.json
* 01:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 01:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 01:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26245 and previous config saved to /var/cache/conftool/dbconfig/20220423-013336-ladsgroup.json
* 01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26244 and previous config saved to /var/cache/conftool/dbconfig/20220423-011945-ladsgroup.json
* 01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26243 and previous config saved to /var/cache/conftool/dbconfig/20220423-010440-ladsgroup.json
* 00:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 00:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 00:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 00:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 00:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 00:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 00:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26242 and previous config saved to /var/cache/conftool/dbconfig/20220423-005613-ladsgroup.json
* 00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26241 and previous config saved to /var/cache/conftool/dbconfig/20220423-004935-ladsgroup.json
* 00:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26240 and previous config saved to /var/cache/conftool/dbconfig/20220423-004617-ladsgroup.json
* 00:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 00:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 00:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26239 and previous config saved to /var/cache/conftool/dbconfig/20220423-004108-ladsgroup.json
* 00:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26238 and previous config saved to /var/cache/conftool/dbconfig/20220423-002603-ladsgroup.json
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26237 and previous config saved to /var/cache/conftool/dbconfig/20220423-002352-ladsgroup.json
* 00:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 00:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26236 and previous config saved to /var/cache/conftool/dbconfig/20220423-002344-ladsgroup.json
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26235 and previous config saved to /var/cache/conftool/dbconfig/20220423-001058-ladsgroup.json
* 00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26234 and previous config saved to /var/cache/conftool/dbconfig/20220423-000839-ladsgroup.json
== 2022-04-22 ==
* 23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26233 and previous config saved to /var/cache/conftool/dbconfig/20220422-235334-ladsgroup.json
* 23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26232 and previous config saved to /var/cache/conftool/dbconfig/20220422-233829-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26231 and previous config saved to /var/cache/conftool/dbconfig/20220422-232210-ladsgroup.json
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 23:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26230 and previous config saved to /var/cache/conftool/dbconfig/20220422-232147-ladsgroup.json
* 23:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26229 and previous config saved to /var/cache/conftool/dbconfig/20220422-230642-ladsgroup.json
* 22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26228 and previous config saved to /var/cache/conftool/dbconfig/20220422-225735-ladsgroup.json
* 22:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 22:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26227 and previous config saved to /var/cache/conftool/dbconfig/20220422-225136-ladsgroup.json
* 22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26226 and previous config saved to /var/cache/conftool/dbconfig/20220422-223631-ladsgroup.json
* 22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26225 and previous config saved to /var/cache/conftool/dbconfig/20220422-222203-ladsgroup.json
* 22:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26224 and previous config saved to /var/cache/conftool/dbconfig/20220422-220658-ladsgroup.json
* 21:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26223 and previous config saved to /var/cache/conftool/dbconfig/20220422-215153-ladsgroup.json
* 21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26222 and previous config saved to /var/cache/conftool/dbconfig/20220422-213648-ladsgroup.json
* 21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26221 and previous config saved to /var/cache/conftool/dbconfig/20220422-213617-ladsgroup.json
* 21:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26220 and previous config saved to /var/cache/conftool/dbconfig/20220422-213609-ladsgroup.json
* 21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26219 and previous config saved to /var/cache/conftool/dbconfig/20220422-212104-ladsgroup.json
* 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26218 and previous config saved to /var/cache/conftool/dbconfig/20220422-210559-ladsgroup.json
* 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26217 and previous config saved to /var/cache/conftool/dbconfig/20220422-205538-ladsgroup.json
* 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26216 and previous config saved to /var/cache/conftool/dbconfig/20220422-205053-ladsgroup.json
* 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26215 and previous config saved to /var/cache/conftool/dbconfig/20220422-204547-ladsgroup.json
* 20:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 20:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26214 and previous config saved to /var/cache/conftool/dbconfig/20220422-204033-ladsgroup.json
* 20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26213 and previous config saved to /var/cache/conftool/dbconfig/20220422-202903-ladsgroup.json
* 20:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 20:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 20:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 20:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26212 and previous config saved to /var/cache/conftool/dbconfig/20220422-202528-ladsgroup.json
* 20:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26211 and previous config saved to /var/cache/conftool/dbconfig/20220422-201023-ladsgroup.json
* 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26210 and previous config saved to /var/cache/conftool/dbconfig/20220422-200605-ladsgroup.json
* 20:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 20:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 20:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26209 and previous config saved to /var/cache/conftool/dbconfig/20220422-191935-ladsgroup.json
* 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26208 and previous config saved to /var/cache/conftool/dbconfig/20220422-191820-ladsgroup.json
* 19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26207 and previous config saved to /var/cache/conftool/dbconfig/20220422-191812-ladsgroup.json
* 19:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 19:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26206 and previous config saved to /var/cache/conftool/dbconfig/20220422-190632-ladsgroup.json
* 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26205 and previous config saved to /var/cache/conftool/dbconfig/20220422-190306-ladsgroup.json
* 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26204 and previous config saved to /var/cache/conftool/dbconfig/20220422-185126-ladsgroup.json
* 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26203 and previous config saved to /var/cache/conftool/dbconfig/20220422-184801-ladsgroup.json
* 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26202 and previous config saved to /var/cache/conftool/dbconfig/20220422-183621-ladsgroup.json
* 18:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26201 and previous config saved to /var/cache/conftool/dbconfig/20220422-183256-ladsgroup.json
* 18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26200 and previous config saved to /var/cache/conftool/dbconfig/20220422-182116-ladsgroup.json
* 17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26199 and previous config saved to /var/cache/conftool/dbconfig/20220422-173242-ladsgroup.json
* 17:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 17:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26198 and previous config saved to /var/cache/conftool/dbconfig/20220422-173234-ladsgroup.json
* 17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26197 and previous config saved to /var/cache/conftool/dbconfig/20220422-173031-ladsgroup.json
* 17:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance