You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26967 and previous config saved to /var/cache/conftool/dbconfig/20220429-005702-ladsgroup.json)
imported>Stashbot
(mutante: wikistats-bullseye:~$ /usr/lib/wikistats/update.php wp prefix blk ; /usr/lib/wikistats/update.php wp prefix kcg T315121)
(104 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== 2022-04-29 ==
== 2022-08-12 ==
* 00:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26967 and previous config saved to /var/cache/conftool/dbconfig/20220429-005702-ladsgroup.json
* 23:41 mutante: wikistats-bullseye:~$ /usr/lib/wikistats/update.php wp prefix blk ; /usr/lib/wikistats/update.php wp prefix kcg [[phab:T315121|T315121]]
* 00:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2008.mgmt.codfw.wmnet with reboot policy FORCED
* 23:38 mutante: [mwmaint1002:~] $ sudo systemctl start mediawiki_job_initsitestats.timer [[phab:T315121|T315121]]
* 00:56 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 22:14 ryankemper@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 00:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26966 and previous config saved to /var/cache/conftool/dbconfig/20220429-005528-ladsgroup.json
* 21:48 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1071.eqiad.wmnet with OS bullseye
* 00:53 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 21:45 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb2002-dev.codfw.wmnet with OS bullseye
* 00:47 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2007.mgmt.codfw.wmnet with reboot policy FORCED
* 21:27 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1071.eqiad.wmnet with reason: host reimage
* 00:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2005.mgmt.codfw.wmnet with reboot policy FORCED
* 21:25 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1071.eqiad.wmnet with reason: host reimage
* 00:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 21:12 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1071.eqiad.wmnet with OS bullseye
* 00:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26965 and previous config saved to /var/cache/conftool/dbconfig/20220429-004157-ladsgroup.json
* 21:10 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb2002-dev.codfw.wmnet with reason: host reimage
* 00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26964 and previous config saved to /var/cache/conftool/dbconfig/20220429-004023-ladsgroup.json
* 21:06 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb2002-dev.codfw.wmnet with reason: host reimage
* 00:38 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 21:06 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1053.eqiad.wmnet with OS bullseye
* 00:37 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs2006.mgmt.codfw.wmnet with reboot policy FORCED
* 20:50 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host clouddb2002-dev.codfw.wmnet with OS bullseye
* 00:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26963 and previous config saved to /var/cache/conftool/dbconfig/20220429-002518-ladsgroup.json
* 20:43 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1053.eqiad.wmnet with reason: host reimage
* 00:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26962 and previous config saved to /var/cache/conftool/dbconfig/20220429-001840-ladsgroup.json
* 20:39 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1053.eqiad.wmnet with reason: host reimage
* 00:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 20:24 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1053.eqiad.wmnet with OS bullseye
* 00:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 20:12 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1048.eqiad.wmnet with OS bullseye
* 00:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26961 and previous config saved to /var/cache/conftool/dbconfig/20220429-001832-ladsgroup.json
* 19:55 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1048.eqiad.wmnet with reason: host reimage
* 00:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26960 and previous config saved to /var/cache/conftool/dbconfig/20220429-001333-ladsgroup.json
* 19:53 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1048.eqiad.wmnet with reason: host reimage
* 00:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 19:42 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1048.eqiad.wmnet with OS bullseye
* 00:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 19:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32375 and previous config saved to /var/cache/conftool/dbconfig/20220812-193822-ladsgroup.json
* 00:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26959 and previous config saved to /var/cache/conftool/dbconfig/20220429-001320-ladsgroup.json
* 19:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 00:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26958 and previous config saved to /var/cache/conftool/dbconfig/20220429-000327-ladsgroup.json
* 19:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 19:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32374 and previous config saved to /var/cache/conftool/dbconfig/20220812-193801-ladsgroup.json
* 19:33 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1054.eqiad.wmnet with OS bullseye
* 19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P32373 and previous config saved to /var/cache/conftool/dbconfig/20220812-192255-ladsgroup.json
* 19:12 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1054.eqiad.wmnet with reason: host reimage
* 19:09 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1054.eqiad.wmnet with reason: host reimage
* 19:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P32372 and previous config saved to /var/cache/conftool/dbconfig/20220812-190749-ladsgroup.json
* 18:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet with reason: Maint
* 18:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet with reason: Maint
* 18:54 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1054.eqiad.wmnet with OS bullseye
* 18:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32371 and previous config saved to /var/cache/conftool/dbconfig/20220812-185243-ladsgroup.json
* 18:48 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1066.eqiad.wmnet with OS bullseye
* 18:25 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1066.eqiad.wmnet with reason: host reimage
* 18:22 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1066.eqiad.wmnet with reason: host reimage
* 18:08 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1066.eqiad.wmnet with OS bullseye
* 18:00 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1064.eqiad.wmnet with OS bullseye
* 17:42 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1064.eqiad.wmnet with reason: host reimage
* 17:39 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1064.eqiad.wmnet with reason: host reimage
* 17:24 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1064.eqiad.wmnet with OS bullseye
* 17:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts netmon2002.wikimedia.org
* 17:21 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts netmon2002.wikimedia.org
* 17:19 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netmon2002.wikimedia.org with OS bullseye
* 17:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netmon2002.wikimedia.org with reason: host reimage
* 17:01 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on netmon2002.wikimedia.org with reason: host reimage
* 16:42 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host netmon2002.wikimedia.org with OS bullseye
* 16:26 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1067.eqiad.wmnet with OS bullseye
* 16:21 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcontrol2003-dev.wikimedia.org
* 16:21 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:16 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 16:11 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudcontrol2003-dev.wikimedia.org
* 16:08 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['netmon2002.wikimedia.org']
* 16:03 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1067.eqiad.wmnet with reason: host reimage
* 15:58 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1067.eqiad.wmnet with reason: host reimage
* 15:43 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1067.eqiad.wmnet with OS bullseye
* 15:37 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['netmon2002.wikimedia.org']
* 15:31 jbond@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['netmon2002.wikimedia.org']
* 15:31 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['netmon2002.wikimedia.org']
* 15:07 jbond@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts netmon1002.wikimedia.org
* 15:07 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts netmon1002.wikimedia.org
* 15:04 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1061.eqiad.wmnet with OS bullseye
* 14:46 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1061.eqiad.wmnet with reason: host reimage
* 14:46 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2042.codfw.wmnet,service=varnish-fe
* 14:46 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2042.codfw.wmnet,service=ats-be
* 14:46 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2042.codfw.wmnet,service=ats-tls
* 14:43 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1061.eqiad.wmnet with reason: host reimage
* 14:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maint
* 14:28 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1061.eqiad.wmnet with OS bullseye
* 14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maint
* 14:24 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1063.eqiad.wmnet with OS bullseye
* 14:05 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1063.eqiad.wmnet with reason: host reimage
* 14:02 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1063.eqiad.wmnet with reason: host reimage
* 13:47 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1063.eqiad.wmnet with OS bullseye
* 13:41 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 06:01 ryankemper@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=elastic10[8-9][0-9].*
* 05:54 ryankemper@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=elastic110.*
* 01:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32369 and previous config saved to /var/cache/conftool/dbconfig/20220812-010312-ladsgroup.json
* 01:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 01:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 01:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 01:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 01:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32368 and previous config saved to /var/cache/conftool/dbconfig/20220812-010233-ladsgroup.json
* 00:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P32367 and previous config saved to /var/cache/conftool/dbconfig/20220812-004727-ladsgroup.json
* 00:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P32366 and previous config saved to /var/cache/conftool/dbconfig/20220812-003221-ladsgroup.json
* 00:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32365 and previous config saved to /var/cache/conftool/dbconfig/20220812-001715-ladsgroup.json


== 2022-04-28 ==
== 2022-08-11 ==
* 23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26957 and previous config saved to /var/cache/conftool/dbconfig/20220428-235815-ladsgroup.json
* 21:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:49 dzahn@deploy1002: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 21:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:49 dzahn@deploy1002: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 21:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:48 dzahn@deploy1002: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 21:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26956 and previous config saved to /var/cache/conftool/dbconfig/20220428-234822-ladsgroup.json
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:47 dzahn@deploy1002: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 21:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:44 dzahn@deploy1002: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 21:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26955 and previous config saved to /var/cache/conftool/dbconfig/20220428-234310-ladsgroup.json
* 21:21 mwdebug-deploy@deploy1002: helmfile [eqiad]
* 23:42 dzahn@deploy1002: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 23:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26954 and previous config saved to /var/cache/conftool/dbconfig/20220428-233317-ladsgroup.json
* 23:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26953 and previous config saved to /var/cache/conftool/dbconfig/20220428-233103-ladsgroup.json
* 23:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 23:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 23:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia


== 2022-04-27 ==
== 2022-08-10 ==
* 23:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26846 and previous config saved to /var/cache/conftool/dbconfig/20220427-235953-ladsgroup.json
* 21:25 bking@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1016.eqiad.wmnet
* 23:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26845 and previous config saved to /var/cache/conftool/dbconfig/20220427-235700-ladsgroup.json
* 21:23 bking@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1014.eqiad.wmnet
* 23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26844 and previous config saved to /var/cache/conftool/dbconfig/20220427-235344-ladsgroup.json
* 21:10 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 16 hosts with reason: [[phab:T309810|T309810]]
* 23:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26843 and previous config saved to /var/cache/conftool/dbconfig/20220427-234739-ladsgroup.json
* 21:10 bking@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on 16 hosts with reason: [[phab:T309810|T309810]]
* 23:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26842 and previous config saved to /var/cache/conftool/dbconfig/20220427-234448-ladsgroup.json
* 21:09 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: [[phab:T309810|T309810]]
* 23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26841 and previous config saved to /var/cache/conftool/dbconfig/20220427-234155-ladsgroup.json
* 21:09 bking@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: [[phab:T309810|T309810]]
* 23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26840 and previous config saved to /var/cache/conftool/dbconfig/20220427-233839-ladsgroup.json
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26839 and previous config saved to /var/cache/conftool/dbconfig/20220427-233628-ladsgroup.json
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26838 and previous config saved to /var/cache/conftool/dbconfig/20220427-233541-ladsgroup.json
* 23:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26837 and previous config saved to /var/cache/conftool/dbconfig/20220427-233234-ladsgroup.json
* 23:31 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host gitlab-runner2002.codfw.wmnet with OS bullseye
* 23:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26836 and previous config saved to /var/cache/conftool/dbconfig/20220427-232942-ladsgroup.json
* 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26835 and previous config saved to /var/cache/conftool/dbconfig/20220427-232650-ladsgroup.json
* 23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26834 and previous config saved to /var/cache/conftool/dbconfig/20220427-232036-ladsgroup.json
* 23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26833 and previous config saved to /var/cache/conftool/dbconfig/20220427-231729-ladsgroup.json
* 23:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26832 and previous config saved to /var/cache/conftool/dbconfig/20220427-231437-ladsgroup.json
* 23:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26831 and previous config saved to /var/cache/conftool/dbconfig/20220427-231145-ladsgroup.json
* 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26830 and previous config saved to /var/cache/conftool/dbconfig/20220427-231037-ladsgroup.json
* 23:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26829 and previous config saved to /var/cache/conftool/dbconfig/20220427-231029-ladsgroup.json
* 23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26828 and previous config saved to /var/cache/conftool/dbconfig/20220427-230531-ladsgroup.json
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26827 and previous config saved to /var/cache/conftool/dbconfig/20220427-230130-ladsgroup.json
* 23:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26826 and previous config saved to /var/cache/conftool/dbconfig/20220427-230116-ladsgroup.json
* 22:59 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab-runner2002.codfw.wmnet with OS bullseye
* 22:58 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host gitlab2002.wikimedia.org with OS bullseye
* 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26825 and previous config saved to /var/cache/conftool/dbconfig/20220427-225524-ladsgroup.json
* 22:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26824 and previous config saved to /var/cache/conftool/dbconfig/20220427-225517-ladsgroup.json
* 22:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 22:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26823 and previous config saved to /var/cache/conftool/dbconfig/20220427-225445-ladsgroup.json
* 22:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26822 and previous config saved to /var/cache/conftool/dbconfig/20220427-225026-ladsgroup.json
* 22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26821 and previous config saved to /var/cache/conftool/dbconfig/20220427-224711-ladsgroup.json
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26820 and previous config saved to /var/cache/conftool/dbconfig/20220427-224654-ladsgroup.json
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26819 and previous config saved to /var/cache/conftool/dbconfig/20220427-224610-ladsgroup.json
* 22:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26818 and previous config saved to /var/cache/conftool/dbconfig/20220427-224019-ladsgroup.json
* 22:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26817 and previous config saved to /var/cache/conftool/dbconfig/20220427-223940-ladsgroup.json
* 22:39 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host gitlab2002.wikimedia.org with OS bullseye
* 22:34 mutante: kubernetes - Uprading release=namespaces/namspace-certificates which added developer-portal and image-suggestion namespaces - but only on staging-codfw - ([[phab:T304891|T304891]], [[phab:T305155|T305155]], [[phab:T297140|T297140]])
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26816 and previous config saved to /var/cache/conftool/dbconfig/20220427-223149-ladsgroup.json
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26815 and previous config saved to /var/cache/conftool/dbconfig/20220427-223105-ladsgroup.json
* 22:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26814 and previous config saved to /var/cache/conftool/dbconfig/20220427-222514-ladsgroup.json
* 22:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26813 and previous config saved to /var/cache/conftool/dbconfig/20220427-222435-ladsgroup.json
* 22:24 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
* 22:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26812 and previous config saved to /var/cache/conftool/dbconfig/20220427-222306-ladsgroup.json
* 22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 22:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26811 and previous config saved to /var/cache/conftool/dbconfig/20220427-222230-ladsgroup.json
* 22:19 dzahn@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 22:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2004.mgmt.codfw.wmnet with reboot policy FORCED
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26810 and previous config saved to /var/cache/conftool/dbconfig/20220427-221644-ladsgroup.json
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26809 and previous config saved to /var/cache/conftool/dbconfig/20220427-221600-ladsgroup.json
* 22:12 dzahn@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 22:09 mutante: running puppet on kubemasters - adding namespace to kubernetes for new service image-suggestion ([[phab:T304891|T304891]], [[phab:T305155|T305155]])
* 22:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26808 and previous config saved to /var/cache/conftool/dbconfig/20220427-220930-ladsgroup.json
* 22:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26807 and previous config saved to /var/cache/conftool/dbconfig/20220427-220725-ladsgroup.json
* 22:04 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26806 and previous config saved to /var/cache/conftool/dbconfig/20220427-220139-ladsgroup.json
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26805 and previous config saved to /var/cache/conftool/dbconfig/20220427-215914-ladsgroup.json
* 21:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26804 and previous config saved to /var/cache/conftool/dbconfig/20220427-215828-ladsgroup.json
* 21:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26803 and previous config saved to /var/cache/conftool/dbconfig/20220427-215820-ladsgroup.json
* 21:56 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 21:54 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26802 and previous config saved to /var/cache/conftool/dbconfig/20220427-215220-ladsgroup.json
* 21:50 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 21:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26801 and previous config saved to /var/cache/conftool/dbconfig/20220427-214825-ladsgroup.json
* 21:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 21:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26800 and previous config saved to /var/cache/conftool/dbconfig/20220427-214752-ladsgroup.json
* 21:46 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2412.codfw.wmnet
* 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26799 and previous config saved to /var/cache/conftool/dbconfig/20220427-214315-ladsgroup.json
* 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26798 and previous config saved to /var/cache/conftool/dbconfig/20220427-213715-ladsgroup.json
* 21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26797 and previous config saved to /var/cache/conftool/dbconfig/20220427-213507-ladsgroup.json
* 21:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26796 and previous config saved to /var/cache/conftool/dbconfig/20220427-213458-ladsgroup.json
* 21:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26795 and previous config saved to /var/cache/conftool/dbconfig/20220427-213247-ladsgroup.json
* 21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26794 and previous config saved to /var/cache/conftool/dbconfig/20220427-212810-ladsgroup.json
* 21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26793 and previous config saved to /var/cache/conftool/dbconfig/20220427-211953-ladsgroup.json
* 21:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26792 and previous config saved to /var/cache/conftool/dbconfig/20220427-211742-ladsgroup.json
* 21:16 ebernhardson@deploy1002: Synchronized wmf-config/ProductionServices.php: Config: Revert: [[gerrit:787068{{!}}cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 58s)
* 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26791 and previous config saved to /var/cache/conftool/dbconfig/20220427-211352-ladsgroup.json
* 21:13 ebernhardson@deploy1002: Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:787068{{!}}cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 54s)
* 21:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26790 and previous config saved to /var/cache/conftool/dbconfig/20220427-211305-ladsgroup.json
* 21:12 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787074{{!}}Forward CirrusSearchDeprecation logs to logstash (T218994)]] (duration: 00m 56s)
* 21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26789 and previous config saved to /var/cache/conftool/dbconfig/20220427-211055-ladsgroup.json
* 21:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 21:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26788 and previous config saved to /var/cache/conftool/dbconfig/20220427-211042-ladsgroup.json
* 21:09 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2004.mgmt.codfw.wmnet with reboot policy FORCED
* 21:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2003.mgmt.codfw.wmnet with reboot policy FORCED
* 21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26787 and previous config saved to /var/cache/conftool/dbconfig/20220427-210448-ladsgroup.json
* 21:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26786 and previous config saved to /var/cache/conftool/dbconfig/20220427-210237-ladsgroup.json
* 21:01 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787069{{!}}cirrus: Turn on AB test of wbsearchentities profiles (T306644)]] (duration: 00m 53s)
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26785 and previous config saved to /var/cache/conftool/dbconfig/20220427-210041-ladsgroup.json
* 21:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 21:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 20:59 dzahn@cumin2002: conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1009.eqiad.wmnet
* 20:56 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1009.eqiad.wmnet
* 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26784 and previous config saved to /var/cache/conftool/dbconfig/20220427-205537-ladsgroup.json
* 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26783 and previous config saved to /var/cache/conftool/dbconfig/20220427-204943-ladsgroup.json
* 20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26782 and previous config saved to /var/cache/conftool/dbconfig/20220427-204736-ladsgroup.json
* 20:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 20:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 20:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26781 and previous config saved to /var/cache/conftool/dbconfig/20220427-204728-ladsgroup.json
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26780 and previous config saved to /var/cache/conftool/dbconfig/20220427-204032-ladsgroup.json
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26779 and previous config saved to /var/cache/conftool/dbconfig/20220427-204031-ladsgroup.json
* 20:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 20:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26778 and previous config saved to /var/cache/conftool/dbconfig/20220427-203959-ladsgroup.json
* 20:39 mutante: alert1001 - systemctl start certspotter
* 20:36 mutante: ms-fe1009 - systemctl restart cron
* 20:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
* 20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
* 20:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 20:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 20:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26777 and previous config saved to /var/cache/conftool/dbconfig/20220427-203223-ladsgroup.json
* 20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26776 and previous config saved to /var/cache/conftool/dbconfig/20220427-202527-ladsgroup.json
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26775 and previous config saved to /var/cache/conftool/dbconfig/20220427-202454-ladsgroup.json
* 20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26774 and previous config saved to /var/cache/conftool/dbconfig/20220427-202314-ladsgroup.json
* 20:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:23 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2003.mgmt.codfw.wmnet with reboot policy FORCED
* 20:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26773 and previous config saved to /var/cache/conftool/dbconfig/20220427-202306-ladsgroup.json
* 20:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26772 and previous config saved to /var/cache/conftool/dbconfig/20220427-202234-ladsgroup.json
* 20:21 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab-runner2002.mgmt.codfw.wmnet with reboot policy FORCED
* 20:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26771 and previous config saved to /var/cache/conftool/dbconfig/20220427-201718-ladsgroup.json
* 20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26770 and previous config saved to /var/cache/conftool/dbconfig/20220427-200949-ladsgroup.json
* 20:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26769 and previous config saved to /var/cache/conftool/dbconfig/20220427-200800-ladsgroup.json
* 20:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26768 and previous config saved to /var/cache/conftool/dbconfig/20220427-200729-ladsgroup.json
* 20:06 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784582{{!}}Halt the DiscussionTools A/B test (T291873)]] (duration: 00m 51s)
* 20:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26767 and previous config saved to /var/cache/conftool/dbconfig/20220427-200213-ladsgroup.json
* 20:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26766 and previous config saved to /var/cache/conftool/dbconfig/20220427-200006-ladsgroup.json
* 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 20:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 20:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 19:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26765 and previous config saved to /var/cache/conftool/dbconfig/20220427-195953-ladsgroup.json
* 19:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26764 and previous config saved to /var/cache/conftool/dbconfig/20220427-195444-ladsgroup.json
* 19:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26763 and previous config saved to /var/cache/conftool/dbconfig/20220427-195255-ladsgroup.json
* 19:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26762 and previous config saved to /var/cache/conftool/dbconfig/20220427-195224-ladsgroup.json
* 19:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26761 and previous config saved to /var/cache/conftool/dbconfig/20220427-194448-ladsgroup.json
* 19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26760 and previous config saved to /var/cache/conftool/dbconfig/20220427-193749-ladsgroup.json
* 19:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26759 and previous config saved to /var/cache/conftool/dbconfig/20220427-193719-ladsgroup.json
* 19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26758 and previous config saved to /var/cache/conftool/dbconfig/20220427-193233-ladsgroup.json
* 19:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 19:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 19:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26757 and previous config saved to /var/cache/conftool/dbconfig/20220427-193201-ladsgroup.json
* 19:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26756 and previous config saved to /var/cache/conftool/dbconfig/20220427-192943-ladsgroup.json
* 19:28 otto@deploy1002: Finished deploy [airflow-dags/analytics@6684963]: (no justification provided) (duration: 00m 09s)
* 19:28 otto@deploy1002: Started deploy [airflow-dags/analytics@6684963]: (no justification provided)
* 19:24 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 11s)
* 19:24 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 19:23 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 03s)
* 19:23 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26755 and previous config saved to /var/cache/conftool/dbconfig/20220427-192209-ladsgroup.json
* 19:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 19:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 19:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26754 and previous config saved to /var/cache/conftool/dbconfig/20220427-192200-ladsgroup.json
* 19:21 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 03s)
* 19:21 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 19:18 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab-runner2002.mgmt.codfw.wmnet with reboot policy FORCED
* 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26753 and previous config saved to /var/cache/conftool/dbconfig/20220427-191656-ladsgroup.json
* 19:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26752 and previous config saved to /var/cache/conftool/dbconfig/20220427-191438-ladsgroup.json
* 19:08 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab2003.mgmt.codfw.wmnet with reboot policy FORCED
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26751 and previous config saved to /var/cache/conftool/dbconfig/20220427-190655-ladsgroup.json
* 19:02 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1004.eqiad.wmnet
* 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26750 and previous config saved to /var/cache/conftool/dbconfig/20220427-190151-ladsgroup.json
* 19:00 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:00 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26749 and previous config saved to /var/cache/conftool/dbconfig/20220427-185150-ladsgroup.json
* 18:51 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1004.eqiad.wmnet
* 18:50 dancy@deploy1002: Started scap: testing mediawiki container image build and deploy
* 18:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26748 and previous config saved to /var/cache/conftool/dbconfig/20220427-184646-ladsgroup.json
* 18:46 otto@deploy1002: Finished deploy [airflow-dags/analytics_test@6684963]: (no justification provided) (duration: 00m 12s)
* 18:46 otto@deploy1002: Started deploy [airflow-dags/analytics_test@6684963]: (no justification provided)
* 18:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab2003.mgmt.codfw.wmnet with reboot policy FORCED
* 18:45 dancy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:45 dancy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:38 dancy@deploy1002: Started scap: (no justification provided)
* 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26746 and previous config saved to /var/cache/conftool/dbconfig/20220427-183735-ladsgroup.json
* 18:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 18:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 18:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26745 and previous config saved to /var/cache/conftool/dbconfig/20220427-183727-ladsgroup.json
* 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26744 and previous config saved to /var/cache/conftool/dbconfig/20220427-183645-ladsgroup.json
* 18:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gitlab2002.mgmt.codfw.wmnet with reboot policy FORCED
* 18:26 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1003.eqiad.wmnet
* 18:25 brennen@deploy1002: Synchronized php: group1 wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]] (duration: 00m 56s)
* 18:24 brennen@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.9  refs [[phab:T305215|T305215]]
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26743 and previous config saved to /var/cache/conftool/dbconfig/20220427-182238-ladsgroup.json
* 18:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 18:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26742 and previous config saved to /var/cache/conftool/dbconfig/20220427-182230-ladsgroup.json
* 18:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26741 and previous config saved to /var/cache/conftool/dbconfig/20220427-182226-ladsgroup.json
* 18:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26740 and previous config saved to /var/cache/conftool/dbconfig/20220427-182222-ladsgroup.json
* 18:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 18:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 18:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26739 and previous config saved to /var/cache/conftool/dbconfig/20220427-181423-ladsgroup.json
* 18:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 18:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 18:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26738 and previous config saved to /var/cache/conftool/dbconfig/20220427-181415-ladsgroup.json
* 18:14 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1003.eqiad.wmnet
* 18:12 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host gitlab2002.mgmt.codfw.wmnet with reboot policy FORCED
* 18:11 brennen@deploy1002: Synchronized php-1.39.0-wmf.9/skins/WikimediaApiPortal: Backport: [[gerrit:786420{{!}}Fix warnings relating to QuickTemplate (T306925)]] (duration: 02m 26s)
* 18:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26737 and previous config saved to /var/cache/conftool/dbconfig/20220427-180725-ladsgroup.json
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26736 and previous config saved to /var/cache/conftool/dbconfig/20220427-180717-ladsgroup.json
* 18:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:00 brennen: train 1.39.0-wmf.9 ([[phab:T305215|T305215]]): no current blockers, proceeding to group1
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26735 and previous config saved to /var/cache/conftool/dbconfig/20220427-175910-ladsgroup.json
* 17:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26734 and previous config saved to /var/cache/conftool/dbconfig/20220427-175220-ladsgroup.json
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26733 and previous config saved to /var/cache/conftool/dbconfig/20220427-175212-ladsgroup.json
* 17:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26732 and previous config saved to /var/cache/conftool/dbconfig/20220427-175100-ladsgroup.json
* 17:51 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 17:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 17:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:49 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:49 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:49 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:45 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26731 and previous config saved to /var/cache/conftool/dbconfig/20220427-174405-ladsgroup.json
* 17:41 ejegg: updated payments-wiki from {{Gerrit|16c0c111}} to {{Gerrit|41e91033}}
* 17:39 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be1002.eqiad.wmnet
* 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26730 and previous config saved to /var/cache/conftool/dbconfig/20220427-173709-ladsgroup.json
* 17:36 dancy: dancy@deploy1002  Testing image build and deployment
* 17:35 dancy@deploy1002: Started scap: (no justification provided)
* 17:29 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1002.eqiad.wmnet
* 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26729 and previous config saved to /var/cache/conftool/dbconfig/20220427-172900-ladsgroup.json
* 17:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|88ac6b94ff5acfbcd6a539e21b7454b6a38eaaba}}: Enable Wikistories on enwiki beta ([[phab:T303004|T303004]]; 2/2) (duration: 00m 50s)
* 17:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|88ac6b94ff5acfbcd6a539e21b7454b6a38eaaba}}: Enable Wikistories on enwiki beta ([[phab:T303004|T303004]]; 1/2) (duration: 00m 51s)
* 17:27 urbanecm@deploy1002: Synchronized wmf-config/extension-list: {{Gerrit|01dfaf063d14ee329c43d65566270ff3cec48d50}}: Add Wikistories extension ([[phab:T303004|T303004]]) (duration: 00m 49s)
* 17:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26728 and previous config saved to /var/cache/conftool/dbconfig/20220427-172653-ladsgroup.json
* 17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 17:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 17:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26727 and previous config saved to /var/cache/conftool/dbconfig/20220427-172211-ladsgroup.json
* 17:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 17:10 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 17:01 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:58 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 16:56 herron@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host thanos-be1001.eqiad.wmnet
* 16:53 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 16:52 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 16:52 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudnet[2002,2004]-dev.codfw.wmnet
* 16:46 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 16:43 dancy@deploy1002: Finished deploy [releng/phatality@d8e2adc]: (no justification provided) (duration: 00m 13s)
* 16:43 dancy@deploy1002: Started deploy [releng/phatality@d8e2adc]: (no justification provided)
* 16:41 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be1001.eqiad.wmnet
* 16:35 razzi@cumin1001: conftool action : set/pooled=inactive; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 16:34 razzi@cumin1001: conftool action : set/pooled=yes; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 16:28 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 16:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26725 and previous config saved to /var/cache/conftool/dbconfig/20220427-161459-ladsgroup.json
* 16:04 Amir1: foreachwikiindblist s6 mysql.php -- -e "desc querycache_info;" {{!}} grep -i qci_timestamp {{!}} grep -i varbinary {{!}} awk '<nowiki>{</nowiki> print substr($1, 1, length($1)-1) <nowiki>}</nowiki>' {{!}} xargs -I <nowiki>{</nowiki><nowiki>}</nowiki> sql <nowiki>{</nowiki><nowiki>}</nowiki> --write -- -e 'ALTER TABLE /*_*/querycache_info CHANGE qci_timestamp qci_timestamp BINARY(14) DEFAULT '19700101000000' NOT NULL;' ([[phab:T298559|T298559]])
* 15:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26724 and previous config saved to /var/cache/conftool/dbconfig/20220427-155954-ladsgroup.json
* 15:58 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1003.eqiad.wmnet
* 15:56 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 15:56 jayme@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 15:52 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2004.codfw.wmnet
* 15:50 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1003.eqiad.wmnet
* 15:50 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet
* 15:48 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet
* 15:47 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet
* 15:45 dcaro@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet
* 15:45 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@6684963]: (no justification provided) (duration: 00m 08s)
* 15:45 nokafor@deploy1002: Started deploy [airflow-dags/analytics@6684963]: (no justification provided)
* 15:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26723 and previous config saved to /var/cache/conftool/dbconfig/20220427-154449-ladsgroup.json
* 15:39 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2004.codfw.wmnet
* 15:36 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 15:35 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 15:33 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 15:31 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 15:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26722 and previous config saved to /var/cache/conftool/dbconfig/20220427-152944-ladsgroup.json
* 15:02 moritzm: installing mariadb-10.5 updates (as packaged in Debian Bullseye, unrelated to wmf-mariadb packages)
* 14:57 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2003.codfw.wmnet
* 14:54 razzi@cumin1001: conftool action : set/pooled=inactive; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 14:54 razzi@cumin1001: conftool action : set/pooled=no; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 14:53 razzi@cumin1001: conftool action : set/pooled=yes; selector: service=cloudceph,name=cloudcephmon1003.eqiad.wmnet
* 14:47 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2003.codfw.wmnet
* 14:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:43 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786976{{!}}Enable videojs in eswiki (T303785 T248418)]] (duration: 00m 51s)
* 14:42 moritzm: imported cas 6.4.6.3 to apt.wikimedia.org
* 14:27 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 14:23 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 14:22 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2002.codfw.wmnet
* 14:21 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 14:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26720 and previous config saved to /var/cache/conftool/dbconfig/20220427-141215-ladsgroup.json
* 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 14:09 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host thanos-be2002.codfw.wmnet
* 14:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 14:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance
* 14:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 14:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 14:07 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6001.drmrs.wmnet
* 14:03 klausman@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 14:03 klausman@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 14:00 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus6001.drmrs.wmnet
* 13:59 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4001.ulsfo.wmnet
* 13:54 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus4001.ulsfo.wmnet
* 13:53 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5001.eqsin.wmnet
* 13:51 jayme@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:47 jayme@cumin1001: START - Cookbook sre.dns.netbox
* 13:46 moritzm: rebalance ganeti-test after adding new bullseye node  [[phab:T306499|T306499]]
* 13:46 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus5001.eqsin.wmnet
* 13:45 herron@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3001.esams.wmnet
* 13:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26719 and previous config saved to /var/cache/conftool/dbconfig/20220427-134308-ladsgroup.json
* 13:40 herron@cumin1001: START - Cookbook sre.hosts.reboot-single for host prometheus3001.esams.wmnet
* 13:37 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
* 13:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26718 and previous config saved to /var/cache/conftool/dbconfig/20220427-133537-ladsgroup.json
* 13:35 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26717 and previous config saved to /var/cache/conftool/dbconfig/20220427-132802-ladsgroup.json
* 13:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26716 and previous config saved to /var/cache/conftool/dbconfig/20220427-132032-ladsgroup.json
* 13:13 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 13:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26715 and previous config saved to /var/cache/conftool/dbconfig/20220427-131257-ladsgroup.json
* 13:11 mvernon@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2040.codfw.wmnet with OS bullseye
* 13:05 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 13:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26714 and previous config saved to /var/cache/conftool/dbconfig/20220427-130527-ladsgroup.json
* 12:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26712 and previous config saved to /var/cache/conftool/dbconfig/20220427-125752-ladsgroup.json
* 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26710 and previous config saved to /var/cache/conftool/dbconfig/20220427-125427-ladsgroup.json
* 12:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 12:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 12:53 kormat: moved pc1014 to pc2 [[phab:T303174|T303174]]
* 12:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26709 and previous config saved to /var/cache/conftool/dbconfig/20220427-125022-ladsgroup.json
* 12:47 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-be2040.codfw.wmnet with OS bullseye
* 12:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26708 and previous config saved to /var/cache/conftool/dbconfig/20220427-124714-ladsgroup.json
* 12:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 12:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 12:42 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on eqiad
* 12:42 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2010.codfw.wmnet with OS bullseye
* 12:28 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage
* 12:25 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on drmrs
* 12:25 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage
* 12:15 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on esams
* 12:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 12:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 12:07 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS bullseye
* 12:07 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786936{{!}}Set actor migration to READ NEW everywhere (T275246)]] (duration: 00m 53s)
* 11:58 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2011.codfw.wmnet with OS bullseye
* 11:44 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage
* 11:42 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 11:41 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage
* 11:24 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS bullseye
* 11:15 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2012.codfw.wmnet with OS bullseye
* 11:02 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage
* 10:58 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage
* 10:51 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on eqsin
* 10:40 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe2012.codfw.wmnet with OS bullseye
* 10:25 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on codfw
* 10:03 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1010.eqiad.wmnet with OS bullseye
* 09:50 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage
* 09:47 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage
* 09:44 vgutierrez: rolling upgrade trafficserver to 8.0.8-1wm6 on ulsfo
* 09:33 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe1010.eqiad.wmnet with OS bullseye
* 09:30 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1446.eqiad.wmnet
* 09:30 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1445.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1440.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1439.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1438.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1437.eqiad.wmnet
* 09:29 akosiaris@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw1338.eqiad.wmnet
* 09:29 akosiaris: repool the hosts we split off on 2022-04-23 as dedicated videoscalers in the jobrunner cluster. The videoscaling load seems to be normal again.
* 09:22 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1011.eqiad.wmnet with OS bullseye
* 09:11 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage
* 09:08 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage
* 08:54 mvernon@cumin1001: START - Cookbook sre.hosts.reimage for host ms-fe1011.eqiad.wmnet with OS bullseye
* 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26706 and previous config saved to /var/cache/conftool/dbconfig/20220427-084941-ladsgroup.json
* 08:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26705 and previous config saved to /var/cache/conftool/dbconfig/20220427-083436-ladsgroup.json
* 08:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26704 and previous config saved to /var/cache/conftool/dbconfig/20220427-081931-ladsgroup.json
* 08:17 marostegui@cumin1001: dbctl commit (dc=all): 'Increase db1132 weight [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P26703 and previous config saved to /var/cache/conftool/dbconfig/20220427-081727-marostegui.json
* 08:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26702 and previous config saved to /var/cache/conftool/dbconfig/20220427-080425-ladsgroup.json
* 07:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti-test2001.codfw.wmnet with reason: bullseye update
* 07:49 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti-test2001.codfw.wmnet with reason: bullseye update
* 07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26701 and previous config saved to /var/cache/conftool/dbconfig/20220427-073114-ladsgroup.json
* 07:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 07:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26700 and previous config saved to /var/cache/conftool/dbconfig/20220427-073106-ladsgroup.json
* 07:24 moritzm: installing libxml2 security updates
* 07:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26699 and previous config saved to /var/cache/conftool/dbconfig/20220427-071601-ladsgroup.json
* 07:11 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 07:11 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26698 and previous config saved to /var/cache/conftool/dbconfig/20220427-070056-ladsgroup.json
* 06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26697 and previous config saved to /var/cache/conftool/dbconfig/20220427-064551-ladsgroup.json
* 06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26696 and previous config saved to /var/cache/conftool/dbconfig/20220427-061112-ladsgroup.json
* 06:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 06:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26695 and previous config saved to /var/cache/conftool/dbconfig/20220427-061104-ladsgroup.json
* 05:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26694 and previous config saved to /var/cache/conftool/dbconfig/20220427-055559-ladsgroup.json
* 05:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26693 and previous config saved to /var/cache/conftool/dbconfig/20220427-054054-ladsgroup.json
* 05:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26692 and previous config saved to /var/cache/conftool/dbconfig/20220427-052555-ladsgroup.json
* 05:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26691 and previous config saved to /var/cache/conftool/dbconfig/20220427-052549-ladsgroup.json
* 05:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26690 and previous config saved to /var/cache/conftool/dbconfig/20220427-051050-ladsgroup.json
* 04:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P26689 and previous config saved to /var/cache/conftool/dbconfig/20220427-045545-ladsgroup.json
* 04:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26688 and previous config saved to /var/cache/conftool/dbconfig/20220427-045153-ladsgroup.json
* 04:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 04:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1168 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26687 and previous config saved to /var/cache/conftool/dbconfig/20220427-044040-ladsgroup.json
* 04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1168 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26686 and previous config saved to /var/cache/conftool/dbconfig/20220427-043732-ladsgroup.json
* 04:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 04:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26685 and previous config saved to /var/cache/conftool/dbconfig/20220427-043724-ladsgroup.json
* 04:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26684 and previous config saved to /var/cache/conftool/dbconfig/20220427-042219-ladsgroup.json
* 04:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 04:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 04:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P26683 and previous config saved to /var/cache/conftool/dbconfig/20220427-040714-ladsgroup.json
* 04:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 04:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 03:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 03:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1165 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26682 and previous config saved to /var/cache/conftool/dbconfig/20220427-035208-ladsgroup.json
* 03:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1165 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26681 and previous config saved to /var/cache/conftool/dbconfig/20220427-035001-ladsgroup.json
* 03:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 03:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 03:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 03:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 03:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26680 and previous config saved to /var/cache/conftool/dbconfig/20220427-034948-ladsgroup.json
* 03:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26679 and previous config saved to /var/cache/conftool/dbconfig/20220427-033443-ladsgroup.json
* 03:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 03:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P26678 and previous config saved to /var/cache/conftool/dbconfig/20220427-031938-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26677 and previous config saved to /var/cache/conftool/dbconfig/20220427-030433-ladsgroup.json
* 03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26676 and previous config saved to /var/cache/conftool/dbconfig/20220427-030225-ladsgroup.json
* 03:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 03:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 03:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26675 and previous config saved to /var/cache/conftool/dbconfig/20220427-030217-ladsgroup.json
* 02:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26674 and previous config saved to /var/cache/conftool/dbconfig/20220427-024712-ladsgroup.json
* 02:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26673 and previous config saved to /var/cache/conftool/dbconfig/20220427-024409-ladsgroup.json
* 02:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P26672 and previous config saved to /var/cache/conftool/dbconfig/20220427-023207-ladsgroup.json
* 02:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26671 and previous config saved to /var/cache/conftool/dbconfig/20220427-022053-ladsgroup.json
* 02:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 02:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 02:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26670 and previous config saved to /var/cache/conftool/dbconfig/20220427-022045-ladsgroup.json
* 02:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26669 and previous config saved to /var/cache/conftool/dbconfig/20220427-021702-ladsgroup.json
* 02:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26668 and previous config saved to /var/cache/conftool/dbconfig/20220427-021450-ladsgroup.json
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 02:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26667 and previous config saved to /var/cache/conftool/dbconfig/20220427-021405-ladsgroup.json
* 02:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26666 and previous config saved to /var/cache/conftool/dbconfig/20220427-020540-ladsgroup.json
* 01:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26665 and previous config saved to /var/cache/conftool/dbconfig/20220427-015900-ladsgroup.json
* 01:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26664 and previous config saved to /var/cache/conftool/dbconfig/20220427-015035-ladsgroup.json
* 01:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26663 and previous config saved to /var/cache/conftool/dbconfig/20220427-014355-ladsgroup.json
* 01:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26662 and previous config saved to /var/cache/conftool/dbconfig/20220427-013530-ladsgroup.json
* 01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26661 and previous config saved to /var/cache/conftool/dbconfig/20220427-012850-ladsgroup.json
* 01:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26660 and previous config saved to /var/cache/conftool/dbconfig/20220427-012538-ladsgroup.json
* 01:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26659 and previous config saved to /var/cache/conftool/dbconfig/20220427-012530-ladsgroup.json
* 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26658 and previous config saved to /var/cache/conftool/dbconfig/20220427-011025-ladsgroup.json
* 01:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26657 and previous config saved to /var/cache/conftool/dbconfig/20220427-010001-ladsgroup.json
* 01:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 01:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26656 and previous config saved to /var/cache/conftool/dbconfig/20220427-005953-ladsgroup.json
* 00:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26655 and previous config saved to /var/cache/conftool/dbconfig/20220427-005520-ladsgroup.json
* 00:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26654 and previous config saved to /var/cache/conftool/dbconfig/20220427-004448-ladsgroup.json
* 00:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26653 and previous config saved to /var/cache/conftool/dbconfig/20220427-004015-ladsgroup.json
* 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26652 and previous config saved to /var/cache/conftool/dbconfig/20220427-002943-ladsgroup.json
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26651 and previous config saved to /var/cache/conftool/dbconfig/20220427-002432-ladsgroup.json
* 00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26650 and previous config saved to /var/cache/conftool/dbconfig/20220427-001438-ladsgroup.json
* 00:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26649 and previous config saved to /var/cache/conftool/dbconfig/20220427-000927-ladsgroup.json
 
== 2022-04-26 ==
* 23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26648 and previous config saved to /var/cache/conftool/dbconfig/20220426-235422-ladsgroup.json
* 23:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26647 and previous config saved to /var/cache/conftool/dbconfig/20220426-234224-ladsgroup.json
* 23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26646 and previous config saved to /var/cache/conftool/dbconfig/20220426-234000-ladsgroup.json
* 23:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 23:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26645 and previous config saved to /var/cache/conftool/dbconfig/20220426-233953-ladsgroup.json
* 23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26644 and previous config saved to /var/cache/conftool/dbconfig/20220426-233917-ladsgroup.json
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26643 and previous config saved to /var/cache/conftool/dbconfig/20220426-233642-ladsgroup.json
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 14 hosts with reason: Maintenance
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 14 hosts with reason: Maintenance
* 23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26642 and previous config saved to /var/cache/conftool/dbconfig/20220426-233545-ladsgroup.json
* 23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26641 and previous config saved to /var/cache/conftool/dbconfig/20220426-232447-ladsgroup.json
* 23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26640 and previous config saved to /var/cache/conftool/dbconfig/20220426-232040-ladsgroup.json
* 23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26639 and previous config saved to /var/cache/conftool/dbconfig/20220426-230942-ladsgroup.json
* 23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26638 and previous config saved to /var/cache/conftool/dbconfig/20220426-230535-ladsgroup.json
* 22:57 tgr@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/extension.json: Backport: [[gerrit:786366{{!}}Enable SkinAddFooterLinks hook]] (duration: 00m 51s)
* 22:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26637 and previous config saved to /var/cache/conftool/dbconfig/20220426-225437-ladsgroup.json
* 22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26636 and previous config saved to /var/cache/conftool/dbconfig/20220426-225326-ladsgroup.json
* 22:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26635 and previous config saved to /var/cache/conftool/dbconfig/20220426-225030-ladsgroup.json
* 22:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26634 and previous config saved to /var/cache/conftool/dbconfig/20220426-224757-ladsgroup.json
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26633 and previous config saved to /var/cache/conftool/dbconfig/20220426-224749-ladsgroup.json
* 22:46 tgr@deploy1002: Finished scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]] (duration: 21m 40s)
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26632 and previous config saved to /var/cache/conftool/dbconfig/20220426-223244-ladsgroup.json
* 22:25 tgr@deploy1002: Started scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]]
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26631 and previous config saved to /var/cache/conftool/dbconfig/20220426-221739-ladsgroup.json
* 22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26630 and previous config saved to /var/cache/conftool/dbconfig/20220426-220234-ladsgroup.json
* 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26629 and previous config saved to /var/cache/conftool/dbconfig/20220426-220001-ladsgroup.json
* 22:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 22:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26628 and previous config saved to /var/cache/conftool/dbconfig/20220426-215953-ladsgroup.json
* 21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26627 and previous config saved to /var/cache/conftool/dbconfig/20220426-214448-ladsgroup.json
* 21:38 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9] (duration: 00m 07s)
* 21:37 aqu@deploy1002: Started deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9]
* 21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26626 and previous config saved to /var/cache/conftool/dbconfig/20220426-212943-ladsgroup.json
* 21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26625 and previous config saved to /var/cache/conftool/dbconfig/20220426-211437-ladsgroup.json
* 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26624 and previous config saved to /var/cache/conftool/dbconfig/20220426-211204-ladsgroup.json
* 21:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26623 and previous config saved to /var/cache/conftool/dbconfig/20220426-211156-ladsgroup.json
* 21:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2418.codfw.wmnet
* 21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2417.codfw.wmnet
* 21:02 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2416.codfw.wmnet
* 21:01 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|cab00628da0ba6226ff162cfc848bea35a35783a}}: fix wmgVectorMaxWidthOptionsNamespaces ([[phab:T300182|T300182]]) (duration: 01m 00s)
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:00 cjming: end of UTC late backport window
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:00 mutante: mw2416, mw2417, mw2418 - scap pull
* 20:59 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820533{{!}}Remove unused $wgEnableMWSuggest]] (duration: 03m 04s)
* 20:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26621 and previous config saved to /var/cache/conftool/dbconfig/20220426-205651-ladsgroup.json
* 20:58 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:50 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 07s)
* 20:56 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820568{{!}}Enable new topic tool on dewiki (T313699)]] (duration: 03m 01s)
* 20:50 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
* 20:34 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:822093{{!}}testwiki: set $wgCdnMatchParameterOrder to false (T314868)]] (duration: 03m 20s)
* 20:49 urbanecm@deploy1002: Synchronized wmf-config/SearchSettingsForWikidata.php: {{Gerrit|f76bc806157a3f4c88d44cd467de347b4b471f4e}}: Correct wbsearchentities profiles (duration: 00m 57s)
* 20:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26620 and previous config saved to /var/cache/conftool/dbconfig/20220426-204146-ladsgroup.json
* 20:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:40 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2415.codfw.wmnet
* 20:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2414.codfw.wmnet
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2413.codfw.wmnet
* 20:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:38 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2412.codfw.wmnet
* 20:09 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
* 20:37 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.9/skins/Vector/resources/skins.vector.styles/: {{Gerrit|019a812176bb940383ddeb22f8a74b5d0f447bf1}}: [ToC] Increase threshold for ToC collapsing to 1000px ([[phab:T306904|T306904]]) (duration: 00m 50s)
* 20:36 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/skins/Vector/resources/skins.vector.styles/: {{Gerrit|31ed884d6eda998f8625a88be0f4aa5fd67aef4b}}: [ToC] Increase threshold for ToC collapsing to 1000px ([[phab:T306904|T306904]]) (duration: 00m 50s)
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:33 mutante: mw2412, mw2413, mw2414, mw2415 - scap pull, get into production the first time
* 20:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|fe0e119ef7c768373db4afed21537f85004a8ae2}}: Expand max-width to login, create account, disable on Wikidata ([[phab:T300182|T300182]], [[phab:T306834|T306834]]; 2/2) (duration: 00m 54s)
* 20:28 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 17s)
* 20:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|fe0e119ef7c768373db4afed21537f85004a8ae2}}: Expand max-width to login, create account, disable on Wikidata ([[phab:T300182|T300182]], [[phab:T306834|T306834]]; 1/2) (duration: 00m 56s)
* 20:27 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
* 20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26619 and previous config saved to /var/cache/conftool/dbconfig/20220426-202641-ladsgroup.json
* 20:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|e3ce97b97c1d83dc4f538040da92a571895cb4d0}}: Enable table of contents a/b test on euwiki and hewiki, enable reading depth ([[phab:T306606|T306606]]) (duration: 00m 52s)
* 20:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26618 and previous config saved to /var/cache/conftool/dbconfig/20220426-202407-ladsgroup.json
* 20:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 20:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26617 and previous config saved to /var/cache/conftool/dbconfig/20220426-202359-ladsgroup.json
* 20:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26616 and previous config saved to /var/cache/conftool/dbconfig/20220426-201610-ladsgroup.json
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:11 urbanecm@deploy1002: Synchronized wmf-config/: {{Gerrit|9805e61f7006edf45199a3e22494945bffaaeb4d}}: Add wbsearchentities profiles for testing ([[phab:T306644|T306644]]) (duration: 00m 53s)
* 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26615 and previous config saved to /var/cache/conftool/dbconfig/20220426-200854-ladsgroup.json
* 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:08 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820646{{!}}Start writing to cuc_actor everywhere except s4 and s8 (T233004)]] (duration: 03m 15s)
* 20:05 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|080b8fc573d9d682038e09a7a7ad875bce478c00}}: cirrus: Turn on retry_on_conflict quirk (duration: 00m 53s)
* 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26614 and previous config saved to /var/cache/conftool/dbconfig/20220426-200105-ladsgroup.json
* 19:51 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for mc[2053-2054].codfw.wmnet
* 19:54 dzahn@cumin2002: conftool action : set/pooled=yes; selector: dc=codfw,name=mw2419.codfw.wmnet
* 19:51 rzl@cumin1001: START - Cookbook sre.hosts.remove-downtime for mc[2053-2054].codfw.wmnet
* 19:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26612 and previous config saved to /var/cache/conftool/dbconfig/20220426-195349-ladsgroup.json
* 19:35 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse[2019-2020].codfw.wmnet
* 19:48 mutante: mw2419 - set weight to 25 in conftool, scap pull, first time in production, jobrunner/videoscaler [[phab:T290192|T290192]]
* 19:35 rzl@cumin1001: START - Cookbook sre.hosts.remove-downtime for parse[2019-2020].codfw.wmnet
* 19:46 dzahn@cumin2002: conftool action : set/weight=25; selector: dc=codfw,name=mw2419.codfw.wmnet
* 19:35 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse[2016-2018].codfw.wmnet
* 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26611 and previous config saved to /var/cache/conftool/dbconfig/20220426-194600-ladsgroup.json
* 19:35 rzl@cumin1001: START - Cookbook sre.
* 19:45 dzahn@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2419.codfw.wmnet
* 19:42 aqu@deploy1002: Finished deploy [analytics/refinery@96a3934] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@96a3934] (duration: 07m 19s)
* 19:38 ladsgroup@cumin1001: dbctl commit (dc=all
* 02:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 02:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
 
== 2022-08-09 ==
* 23:17 bking@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1011.eqiad.wmnet
* 23:07 bking@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 23:06 bking@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 22:51 bking@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 22:51 bking@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 22:49 bking@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 22:49 bking@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 22:46 bking@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1015.eqiad.wmnet
* 22:31 ryankemper@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 22:31 ryankemper@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 22:28 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
* 22:02 bking@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 22:02 bking@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 21:57 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs2006.codfw.wmnet with reason: [[phab:T310146|T310146]]
* 21:57 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs2006.codfw.wmnet with reason: [[phab:T310146|T310146]]
* 21:53 bking@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 21:52 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 21:50 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 21:49 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 21:43 bking@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 21:43 bking@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 21:43 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 21:43 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 21:43 bking@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 21:43 bking@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply
* 21:08 bking@cumin1001: START - Cookbook sre.wdqs.data-transfer
* 21:00 bking@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1014.eqiad.wmnet
* 20:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 20:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32332 and previous config saved to /var/cache/conftool/dbconfig/20220809-205548-ladsgroup.json
* 20:51 bking@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs1014.eqiad.wmnet
* 20:51 bking@cumin1001: START - Cookbook sre.hosts.remove-downtime for wdqs1014.eqiad.wmnet
* 20:46 bking@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P32331 and previous config saved to /var/cache/conftool/dbconfig/20220809-204042-ladsgroup.json
* 20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P32330 and previous config saved to /var/cache/conftool/dbconfig/20220809-202536-ladsgroup.json
* 20:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32329 and previous config saved to /var/cache/conftool/dbconfig/20220809-201030-ladsgroup.json
* 19:57 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on wdqs1014.eqiad.wmnet with reason: [[phab:T314890|T314890]]
* 19:57 bking@cumin1001: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on wdqs1014.eqiad.wmnet with reason: [[phab:T314890|T314890]]
* 19:56 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on wdqs1016.eqiad.wmnet with reason: [[phab:T314890|T314890]]
* 19:56 bking@cumin1001: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on wdqs1016.eqiad.wmnet with reason: [[phab:T314890|T314890]]
* 19:55 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T314890|T314890]]
* 19:55 bking@cumin1001: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: [[phab:T314890|T314890]]
* 19:38 dcausse@deploy1002: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
* 19:36 dcausse@deploy1002: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
* 19:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 19:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 19:25 bking@cumin1001: START - Cookbook sre.wdqs.data-transfer
* 18:09 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:06 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:54 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:47 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:38 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1072.eqiad.wmnet with OS bullseye
* 17:29 vgutierrez: test trafficserver 9.1.2-1wm2 in cp6016 - [[phab:T309651|T309651]]
* 17:15 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1072.eqiad.wmnet with reason: host reimage
* 17:13 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1072.eqiad.wmnet with reason: host reimage
* 17:00 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1072.eqiad.wmnet with OS bullseye
* 16:54 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0)
* 16:54 bking@cumin1001: START - Cookbook sre.elasticsearch.force-shard-allocation
* 16:53 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0)
* 16:53 bking@cumin1001: START - Cookbook sre.elasticsearch.force-shard-allocation
* 16:26 bking@deploy1002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 16:26 bking@deploy1002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 16:01 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1069.eqiad.wmnet with OS bullseye
* 15:45 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1069.eqiad.wmnet with reason: host reimage
* 15:42 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1069.eqiad.wmnet with reason: host reimage
* 15:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:30 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1069.eqiad.wmnet with OS bullseye
* 15:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 15:27 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1058.eqiad.wmnet with OS bullseye
* 15:08 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1058.eqiad.wmnet with reason: host reimage
* 15:05 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1058.eqiad.wmnet with reason: host reimage
* 14:59 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 14:59 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* m: finished running 'homer "status:active" commit "netmon: Add the netmon1003 host as a syslog destination"' in the cumin1001 host. Homer reported no errors.
* 14:54 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 14:50 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1058.eqiad.wmnet with OS bullseye
* 14:28 bking@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=codfw
* 13:57 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 13:57 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* m: Add the new netmon1003 host as a syslog destination in homer templates/common/system.conf https://gerrit.wikimedia.org/r/c/operations/homer/public/+/819124
* m: Successfully ran '# run-puppet-merge' in the netmon1002 and netmon1003 hosts.
* m: Running '# run-puppet-agent' in the netmon1003 host
* m: Running '# run-puppet-agent' in the netmon1002 host
* 13:47 ryankemper@cumin1001: END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0)
* 13:46 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.force-shard-allocation
* m: puppet-merge on puppetmaster2004.codfw.wmnet for patch 819179 succeeded
* m: Set netmon1003 as netmon_server and netmon1002 as a netmon_servers_failover in the Puppet repository https://gerrit.wikimedia.org/r/c/operations/puppet/+/819179
* m: authdns updated successfully
* m: Had to revert https://gerrit.wikimedia.org/r/c/operations/dns/+/819177 because I rebased my changes incorrectly, sent the new patch in https://gerrit.wikimedia.org/r/c/operations/dns/+/821746
* m: running '# authdns-update' in  ns0.wikimedia.org
* m: Flip DNS for LibreNMS and Smokeping from netmon1002 to netmon1003 https://gerrit.wikimedia.org/r/c/operations/dns/+/819177
* 13:23 jynus: stop replication on db1117:m1 [[phab:T309074|T309074]]
* m: netmon1002 to netmon1003 failover
* 13:17 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 13:16 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 10:58 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 09:53 vgutierrez: rolling restart of pybal in eqsin - [[phab:T310070|T310070]]
* 09:25 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:24 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:24 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:12 vgutierrez: rolling restart of pybal in codfw - [[phab:T310070|T310070]]
* 08:47 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:30 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:28 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 08:26 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 08:24 jynus: starting data check using es1021 and es2021, expect increased read traffic [[phab:T314559|T314559]]
* 08:21 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 06:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 06:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 06:19 Amir1: dbmaint s5@eqiad ([[phab:T312863|T312863]] [[phab:T312984|T312984]] [[phab:T310011|T310011]] [[phab:T310485|T310485]])
* 06:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1130.eqiad.wmnet with reason: Maint
* 06:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1130.eqiad.wmnet with reason: Maint
* 06:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool db1130 [[phab:T314370|T314370]]', diff saved to https://phabricator.wikimedia.org/P32323 and previous config saved to /var/cache/conftool/dbconfig/20220809-060836-ladsgroup.json
* 06:07 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 06:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Promote db1100 to s5 primary and set section read-write [[phab:T314370|T314370]]', diff saved to https://phabricator.wikimedia.org/P32322 and previous config saved to /var/cache/conftool/dbconfig/20220809-060159-ladsgroup.json
* 06:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - [[phab:T314370|T314370]]', diff saved to https://phabricator.wikimedia.org/P32321 and previous config saved to /var/cache/conftool/dbconfig/20220809-060105-ladsgroup.json
* 06:00 Amir1: Starting s5 eqiad failover from db1130 to db1100 - [[phab:T314370|T314370]]
* 05:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Set db1100 with weight 0 [[phab:T314370|T314370]]', diff saved to https://phabricator.wikimedia.org/P32320 and previous config saved to /var/cache/conftool/dbconfig/20220809-051251-ladsgroup.json
* 05:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T314370|T314370]]
* 05:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 22 hosts with reason: Primary switchover s5 [[phab:T314370|T314370]]
* 02:42 ejegg: SmashPig upgraded from {{Gerrit|9b97ea15}} to {{Gerrit|13e9e9cc}}
* 02:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32318 and previous config saved to /var/cache/conftool/dbconfig/20220809-023113-ladsgroup.json
* 02:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32317 and previous config saved to /var/cache/conftool/dbconfig/20220809-023052-ladsgroup.json
* 02:28 ejegg: payments-wiki upgraded from {{Gerrit|6880236d}} to {{Gerrit|cf5e1848}}
* 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P32316 and previous config saved to /var/cache/conftool/dbconfig/20220809-021546-ladsgroup.json
* 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P32315 and previous config saved to /var/cache/conftool/dbconfig/20220809-020040-ladsgroup.json
* 01:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32314 and previous config saved to /var/cache/conftool/dbconfig/20220809-014534-ladsgroup.json


== 2022-04-25 ==
== 2022-08-08 ==
* 23:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:52 tstarling@deploy1002: Synchronized wmf-config/InitialiseSettings.php: clean up testwiki experiments [[phab:T314750|T314750]] (duration: 03m 19s)
* 23:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:47 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:46 tstarling@deploy1002: Synchronized wmf-config/CommonSettings.php: clean up testwiki experiments [[phab:T314750|T314750]] (duration: 03m 27s)
* 23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 23:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:01 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785948{{!}}ActorMigration: Read from rev_actor field in all of small wikis (T275246)]] (duration: 00m 57s)
* 23:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:45 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:32 eileen___: config revision changed from {{Gerrit|f5668044}} to 787cd0e0<eileen___> eileen
* 22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:32 eileen___: civicrm upgraded from {{Gerrit|497bddf7}} to {{Gerrit|1f91ac2d}}
* 22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:16 ryankemper@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 22:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:16 ryankemper@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host elastic1065.eqiad.wmnet with OS bullseye
* 22:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:53 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1065.eqiad.wmnet with reason: host reimage
* 22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:50 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1065.eqiad.wmnet with reason: host reimage
* 22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:36 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1065.eqiad.wmnet with OS bullseye
* 22:49 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:612348{{!}}TimedMediaHandler: Make videojs the only player on all group1 (T248418)]] (duration: 00m 54s)
* 21:12 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1062.eqiad.wmnet with OS bullseye
* 22:04 dancy@deploy1002: Synchronized README: testing scap mods (duration: 00m 54s)
* 20:53 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1062.eqiad.wmnet with reason: host reimage
* 22:00 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudweb2001-dev.wikimedia.org
* 20:50 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1062.eqiad.wmnet with reason: host reimage
* 21:56 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:36 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1062.eqiad.wmnet with OS bullseye
* 21:52 eileen: civicrm revision {{Gerrit|7de7ddd4}} -> {{Gerrit|a841cf55}}
* 20:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:49 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 20:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:43 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudweb2001-dev.wikimedia.org
* 20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:40 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
* 20:29 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 21:38 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:28 cjming: end of UTC late backport window
* 21:35 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 20:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:26 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
* 20:27 cjming@deploy1002: Synchronized php-1.39.0-wmf.23/skins/Vector/resources/skins.vector.styles/layouts/grid.less: Backport: [[gerrit:821243{{!}}Fix grid blowout bug (T314756)]] (duration: 03m 26s)
* 21:25 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785919{{!}}[Web scroll] Restore original sampling rate (T305442)]] (duration: 01m 01s)
* 20:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:11 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:817785{{!}}Disable sticky header edit A/B test for pilot wikis (T312296)]] (duration: 03m 35s)
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:59 dzahn@cumin2002: conftool action : set/pooled=inactive; selector: dc=codfw,name=mw2286.codfw.wmnet
* 17:34 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1088.eqiad.wmnet with OS bullseye
* 20:58 mutante: rebooting mw2415
* 17:15 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1088.eqiad.wmnet with reason: host reimage
* 20:27 catrope@deploy1002: Synchronized wmf-config/wikitech.php: Config: [[gerrit:785399{{!}}labtestwiki: update labtest ldap server (T304881)]] (duration: 01m 39s)
* 17:12 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1088.eqiad.wmnet with reason: host reimage
* 20:23 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785908{{!}}Enable TOC for all users opted into modern Vector outside of pilot wikis (T306608)]] (duration: 01m 40s)
* 17:00 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1088.eqiad.wmnet with OS bullseye
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:54 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1085.eqiad.wmnet with OS bullseye
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:49 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:43 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:41 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1085.eqiad.wmnet with reason: host reimage
* 20:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:39 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 20:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:38 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1085.eqiad.wmnet with reason: host reimage
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:26 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1085.eqiad.wmnet with OS bullseye
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:24 bking@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host elastic1085.eqiad.wmnet with OS bullseye
* 19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: fresh role user
* 16:19 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1085.eqiad.wmnet with reason: host reimage
* 19:57 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: fresh role user
* 16:16 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1085.eqiad.wmnet with reason: host reimage
* 19:53 mutante: rebooting mw2414 through mw2419
* 16:16 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 19:46 dancy@deploy1002: Finished scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 12m 54s)
* 16:14 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 19:43 mutante: rebooting mw2412, mw2413
* 16:12 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 19:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
* 16:10 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 19:34 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
* 16:09 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 19:33 dancy@deploy1002: Started scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]]
* 16:04 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1085.eqiad.wmnet with OS bullseye
* 19:30 dancy@deploy1002: Started scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]]
* 16:00 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1084.eqiad.wmnet with OS bullseye
* 19:29 dancy@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 01m 43s)
* 15:58 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reimage (bullseye upgrade) - ryankemper@cumin1001 - [[phab:T289135|T289135]]
* 19:27 dancy@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 01m 42s)
* 15:47 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1084.eqiad.wmnet with reason: host reimage
* 19:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:46 sukhe: upload reprepro -C main include bullseye-wikimedia python-pynetbox_6.6.0-1+wmf11u1_amd64.changes
* 19:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:45 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1084.eqiad.wmnet with reason: host reimage
* 19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2021.codfw.wmnet with reason: Maint
* 19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 15:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2021.codfw.wmnet with reason: Maint
* 19:09 mutante: turning mw2412 through mw2419 into actual appservers - applying roles for the first time, will cause alerts probably
* 15:32 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1084.eqiad.wmnet with OS bullseye
* 19:09 cwhite: install grafana-plugins 0.5 and restart grafana on grafana1002 [[phab:T304583|T304583]]
* 14:59 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 18:47 cstone: payments-wiki revision changed from {{Gerrit|a3c69385}} to {{Gerrit|786dc94f}}
* 14:55 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26497 and previous config saved to /var/cache/conftool/dbconfig/20220425-175957-ladsgroup.json
* 14:47 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp5001.eqsin.wmnet with reason: depooled: faulty DIMM: [[phab:T314256|T314256]]
* 17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26496 and previous config saved to /var/cache/conftool/dbconfig/20220425-174451-ladsgroup.json
* 14:46 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on cp5001.eqsin.wmnet with reason: depooled: faulty DIMM: [[phab:T314256|T314256]]
* 17:39 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudservices[2002-2003]-dev.wikimedia.org
* 14:34 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 17:34 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:11 kevinbazira@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 17:30 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 13:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26495 and previous config saved to /var/cache/conftool/dbconfig/20220425-172946-ladsgroup.json
* 13:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:24 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudservices[2002-2003]-dev.wikimedia.org
* 13:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:17 aokoth@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM otrs1001.eqiad.wmnet
* 12:58 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26494 and previous config saved to /var/cache/conftool/dbconfig/20220425-171441-ladsgroup.json
* 12:56 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|77fd5abdd7d9462869259e1511bbcf2d7ce62246}}: Growth: Add new rights to wgAvailableRights (duration: 03m 24s)
* 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26493 and previous config saved to /var/cache/conftool/dbconfig/20220425-171223-ladsgroup.json
* 12:30 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1102.eqiad.wmnet
* 17:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 12:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 12:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:04 aokoth@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM otrs1001.eqiad.wmnet
* 12:06 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.23/extensions/GrowthExperiments/: {{Gerrit|3eaf155678b7313c55dcca0cd39ab29f73eead37}}: MentorTools: Do not use MentorWeightManager ([[phab:T314362|T314362]]) (duration: 03m 31s)
* 16:47 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-eqiad cluster: Reboot kafka nodes
* 12:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 15:56 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
* 11:43 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1102.eqiad.wmnet
* 15:56 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
* 11:21 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=kubernetes2022.codfw.wmnet
* 15:54 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1001.eqiad.wmnet
* 11:21 jelto: kubectl uncordon kubernetes2022.codfw.wmnet
* 15:47 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki1001.eqiad.wmnet
* 10:43 Amir1: Removing db2079 from orchestrator ([[phab:T313885|T313885]])
* 15:46 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
* 10:39 Amir1: Removing db2079 from zarcillo ([[phab:T313885|T313885]])
* 15:46 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
* 10:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2079.codfw.wmnet
* 15:41 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:37 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-eqiad cluster: Reboot kafka nodes
* 10:30 ladsgroup@cumin1001: START - Cookbook sre.dns.netbox
* 15:36 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 10:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.decommission for hosts db2079.codfw.wmnet
* 15:32 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-codfw cluster: Reboot kafka nodes
* 10:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 12:00:00 on db2079.codfw.wmnet with reason: Decom
* 15:25 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 10:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on db2079.codfw.wmnet with reason: Decom
* 15:23 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:41 jbond: deploy libtirpc update
* 15:21 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32310 and previous config saved to /var/cache/conftool/dbconfig/20220808-075723-ladsgroup.json
* 15:20 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1001.eqiad.wmnet
* 07:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 15:18 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2001.codfw.wmnet
* 07:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 15:16 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki-root1001.eqiad.wmnet
* 07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32309 and previous config saved to /var/cache/conftool/dbconfig/20220808-075702-ladsgroup.json
* 15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 07:53 godog: grow sda/sdb 3 by 100G on thanos-be2001 - [[phab:T314275|T314275]]
* 15:15 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
* 07:50 godog: grow sda/sdb 3 by 100G on thanos-be1004 - [[phab:T314275|T314275]]
* 15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 07:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P32308 and previous config saved to /var/cache/conftool/dbconfig/20220808-074156-ladsgroup.json
* 15:15 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 07:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:13 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
* 07:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:12 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 07:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:10 jbond@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "test sync - jbond@cumin1001"
* 07:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P32307 and previous config saved to /var/cache/conftool/dbconfig/20220808-072650-ladsgroup.json
* 15:09 jbond@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test sync - jbond@cumin1001"
* 07:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:52 krinkle@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|I22240af06d}} (duration: 01m 42s)
* 07:22 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820815{{!}}trwikivoyage: Create rollbacker user group (T314678)]] (duration: 03m 17s)
* 14:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:21 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-codfw cluster: Reboot kafka nodes
* 07:11 elukey: restart rsyslog on ml-serve2007
* 14:13 jelto: mw2253: remove puppet lock of stuck puppet run due to reboot, run-puppet-agent
* 14:10 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1011.eqiad.wmnet
* 14:00 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1011.eqiad.wmnet
* 13:51 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1010.eqiad.wmnet
* 13:41 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1010.eqiad.wmnet
* 13:39 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1003.eqiad.wmnet
* 13:35 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1003.eqiad.wmnet
* 13:31 jelto: maintenance (rolling reboot) on api_appserver in codfw (cookbook sre.hosts.reboot-cluster -D codfw -c api_appserver --percentage 5 --grace_sleep 60)
* 13:30 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 13:28 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1002.eqiad.wmnet
* 13:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1002.eqiad.wmnet
* 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26492 and previous config saved to /var/cache/conftool/dbconfig/20220425-131411-ladsgroup.json
* 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:13 urbanecm: UTC afternoon B&C window done
* 13:12 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/CentralAuth/includes/User/GlobalUserSelectQueryBuilder.php: {{Gerrit|c4c4c3219ad69705c83caa50754d95285c96f352}}: GlobalUserSelectQueryBuilder: Do not fatal when no users are returned ([[phab:T306535|T306535]]) (duration: 00m 54s)
* 13:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:04 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0338c9bb9ff6ad388270d1e2df5013d8a49d1210}}: GrowthExperiments: Do not use facebook in campaign pattern ([[phab:T303785|T303785]]) (duration: 00m 51s)
* 13:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2001.wikimedia.org
* 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26491 and previous config saved to /var/cache/conftool/dbconfig/20220425-125906-ladsgroup.json
* 12:58 krinkle@deploy1002: Synchronized private/PrivateSettings.php: {{Gerrit|If4d7ea24b4db59}} (duration: 00m 59s)
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2001.wikimedia.org
* 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26490 and previous config saved to /var/cache/conftool/dbconfig/20220425-124401-ladsgroup.json
* 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26489 and previous config saved to /var/cache/conftool/dbconfig/20220425-122856-ladsgroup.json
* 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26488 and previous config saved to /var/cache/conftool/dbconfig/20220425-122531-ladsgroup.json
* 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26487 and previous config saved to /var/cache/conftool/dbconfig/20220425-122518-ladsgroup.json
* 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26485 and previous config saved to /var/cache/conftool/dbconfig/20220425-121013-ladsgroup.json
* 12:02 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1001.eqiad.wmnet
* 11:58 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1001.eqiad.wmnet
* 11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26484 and previous config saved to /var/cache/conftool/dbconfig/20220425-115508-ladsgroup.json
* 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26483 and previous config saved to /var/cache/conftool/dbconfig/20220425-114003-ladsgroup.json
* 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26482 and previous config saved to /var/cache/conftool/dbconfig/20220425-113138-ladsgroup.json
* 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26481 and previous config saved to /var/cache/conftool/dbconfig/20220425-113130-ladsgroup.json
* 11:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
* 11:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 11:20 moritzm: failover Ganeti master in codfw-test to ganeti-test2003 [[phab:T306499|T306499]]
* 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26480 and previous config saved to /var/cache/conftool/dbconfig/20220425-111625-ladsgroup.json
* 11:13 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26479 and previous config saved to /var/cache/conftool/dbconfig/20220425-111315-kormat.json
* 11:11 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host an-master1001.eqiad.wmnet
* 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26478 and previous config saved to /var/cache/conftool/dbconfig/20220425-110119-ladsgroup.json
* 11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1001.eqiad.wmnet
* 10:58 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26477 and previous config saved to /var/cache/conftool/dbconfig/20220425-105811-kormat.json
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26476 and previous config saved to /var/cache/conftool/dbconfig/20220425-104614-ladsgroup.json
* 10:45 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:45 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:43 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26475 and previous config saved to /var/cache/conftool/dbconfig/20220425-104307-kormat.json
* 10:38 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:38 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:28 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26474 and previous config saved to /var/cache/conftool/dbconfig/20220425-102803-kormat.json
* 10:24 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:24 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flowspec1001.eqiad.wmnet
* 10:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host flowspec1001.eqiad.wmnet
* 10:14 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
* 10:13 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:07 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2419.codfw.wmnet
* 10:05 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:04 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:02 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2419.codfw.wmnet
* 10:02 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2418.codfw.wmnet
* 09:56 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2418.codfw.wmnet
* 09:56 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2417.codfw.wmnet
* 09:55 kormat@cumin1001: dbctl commit (dc=all): 'db1137 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26472 and previous config saved to /var/cache/conftool/dbconfig/20220425-095543-kormat.json
* 09:55 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:55 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:51 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2417.codfw.wmnet
* 09:50 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2416.codfw.wmnet
* 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26471 and previous config saved to /var/cache/conftool/dbconfig/20220425-094600-ladsgroup.json
* 09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 09:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26470 and previous config saved to /var/cache/conftool/dbconfig/20220425-094552-ladsgroup.json
* 09:43 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2416.codfw.wmnet
* 09:42 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2415.codfw.wmnet
* 09:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 09:41 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 09:37 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2415.codfw.wmnet
* 09:36 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2414.codfw.wmnet
* 09:31 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2414.codfw.wmnet
* 09:30 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2413.codfw.wmnet
* 09:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26469 and previous config saved to /var/cache/conftool/dbconfig/20220425-093047-ladsgroup.json
* 09:28 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26468 and previous config saved to /var/cache/conftool/dbconfig/20220425-092807-kormat.json
* 09:25 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2413.codfw.wmnet
* 09:23 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2412.codfw.wmnet
* 09:16 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2412.codfw.wmnet
* 09:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26467 and previous config saved to /var/cache/conftool/dbconfig/20220425-091542-ladsgroup.json
* 09:13 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26466 and previous config saved to /var/cache/conftool/dbconfig/20220425-091303-kormat.json
* 09:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26465 and previous config saved to /var/cache/conftool/dbconfig/20220425-090037-ladsgroup.json
* 08:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1002.eqiad.wmnet
* 08:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26464 and previous config saved to /var/cache/conftool/dbconfig/20220425-085822-ladsgroup.json
* 08:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 08:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet
* 08:58 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26463 and previous config saved to /var/cache/conftool/dbconfig/20220425-085759-kormat.json
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26462 and previous config saved to /var/cache/conftool/dbconfig/20220425-085650-ladsgroup.json
* 08:55 vgutierrez: restart varnish and ats on cp2037 to clear daemon restart alerts
* 08:54 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1002.eqiad.wmnet
* 08:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet
* 08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 9 hosts with reason: Rebooting db1154 [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 9 hosts with reason: Rebooting db1154 [[phab:T303174|T303174]]
* 08:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26461 and previous config saved to /var/cache/conftool/dbconfig/20220425-084318-ladsgroup.json
* 08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26460 and previous config saved to /var/cache/conftool/dbconfig/20220425-084256-kormat.json
* 08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26459 and previous config saved to /var/cache/conftool/dbconfig/20220425-084251-kormat.json
* 08:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26458 and previous config saved to /var/cache/conftool/dbconfig/20220425-084145-ladsgroup.json
* 08:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26457 and previous config saved to /var/cache/conftool/dbconfig/20220425-082813-ladsgroup.json
* 08:27 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26456 and previous config saved to /var/cache/conftool/dbconfig/20220425-082747-kormat.json
* 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26455 and previous config saved to /var/cache/conftool/dbconfig/20220425-082640-ladsgroup.json
* 08:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26454 and previous config saved to /var/cache/conftool/dbconfig/20220425-081307-ladsgroup.json
* 08:12 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26453 and previous config saved to /var/cache/conftool/dbconfig/20220425-081244-kormat.json
* 08:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26452 and previous config saved to /var/cache/conftool/dbconfig/20220425-081135-ladsgroup.json
* 08:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26451 and previous config saved to /var/cache/conftool/dbconfig/20220425-080910-ladsgroup.json
* 08:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 08:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26450 and previous config saved to /var/cache/conftool/dbconfig/20220425-080838-ladsgroup.json
* 07:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader2001.codfw.wmnet
* 07:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26449 and previous config saved to /var/cache/conftool/dbconfig/20220425-075801-ladsgroup.json
* 07:57 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26448 and previous config saved to /var/cache/conftool/dbconfig/20220425-075740-kormat.json
* 07:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26447 and previous config saved to /var/cache/conftool/dbconfig/20220425-075333-ladsgroup.json
* 07:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader2001.codfw.wmnet
* 07:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader1001.eqiad.wmnet
* 07:51 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26446 and previous config saved to /var/cache/conftool/dbconfig/20220425-075106-kormat.json
* 07:50 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26445 and previous config saved to /var/cache/conftool/dbconfig/20220425-075045-kormat.json
* 07:50 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 07:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 07:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org
* 07:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader1001.eqiad.wmnet
* 07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26444 and previous config saved to /var/cache/conftool/dbconfig/20220425-074912-ladsgroup.json
* 07:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 07:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26443 and previous config saved to /var/cache/conftool/dbconfig/20220425-074904-ladsgroup.json
* 07:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org
* 07:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26442 and previous config saved to /var/cache/conftool/dbconfig/20220425-073828-ladsgroup.json
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26441 and previous config saved to /var/cache/conftool/dbconfig/20220425-073359-ladsgroup.json
* 07:31 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host seaborgium.wikimedia.org
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26440 and previous config saved to /var/cache/conftool/dbconfig/20220425-072323-ladsgroup.json
* 07:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org
* 07:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26439 and previous config saved to /var/cache/conftool/dbconfig/20220425-072157-ladsgroup.json
* 07:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 07:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26438 and previous config saved to /var/cache/conftool/dbconfig/20220425-072149-ladsgroup.json
* 07:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2005.codfw.wmnet
* 07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26437 and previous config saved to /var/cache/conftool/dbconfig/20220425-071853-ladsgroup.json
* 07:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2005.codfw.wmnet
* 07:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2004.codfw.wmnet
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:11 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784685{{!}}ActorMigration: Start reading from rev_actor field in group0 (T275246)]] (duration: 00m 50s)
* 07:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32306 and previous config saved to /var/cache/conftool/dbconfig/20220808-071144-ladsgroup.json
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:09 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820261{{!}}Enable SectionTranslation on 10 Wikipedias where ContentTranslation is default (T308829)]] (duration: 03m 15s)
* 07:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2004.codfw.wmnet
* 07:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26436 and previous config saved to /var/cache/conftool/dbconfig/20220425-070644-ladsgroup.json
* 07:06 XioNoX: add CSP headers to Netbox - [[phab:T296356|T296356]]
* 07:06 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785326{{!}}Add tothemoon.ser.asu.edu to the wgCopyUploadsDomains allowlist of commonswiki (T306671)]] (duration: 00m 52s)
* 07:05 elukey: restart rsyslog on ml-serve-ctrl2001
* 07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26435 and previous config saved to /var/cache/conftool/dbconfig/20220425-070348-ladsgroup.json
* 06:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1001.wikimedia.org
* 06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26434 and previous config saved to /var/cache/conftool/dbconfig/20220425-065559-ladsgroup.json
* 06:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 06:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 06:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1001.wikimedia.org
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26433 and previous config saved to /var/cache/conftool/dbconfig/20220425-065139-ladsgroup.json
* 06:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1132 into s1 [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P26432 and previous config saved to /var/cache/conftool/dbconfig/20220425-063823-marostegui.json
* 06:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26431 and previous config saved to /var/cache/conftool/dbconfig/20220425-063634-ladsgroup.json
* 06:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26430 and previous config saved to /var/cache/conftool/dbconfig/20220425-063409-ladsgroup.json
* 06:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:11 dcausse: depooling and restarting blazegraph on wdqs1007 (deadlocked for 4+days)
* 04:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26429 and previous config saved to /var/cache/conftool/dbconfig/20220425-045902-ladsgroup.json
* 04:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26428 and previous config saved to /var/cache/conftool/dbconfig/20220425-044357-ladsgroup.json
* 04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26427 and previous config saved to /var/cache/conftool/dbconfig/20220425-042852-ladsgroup.json
* 04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26426 and previous config saved to /var/cache/conftool/dbconfig/20220425-041347-ladsgroup.json
* 04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26425 and previous config saved to /var/cache/conftool/dbconfig/20220425-040940-ladsgroup.json
* 04:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 04:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26424 and previous config saved to /var/cache/conftool/dbconfig/20220425-040926-ladsgroup.json
* 03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26423 and previous config saved to /var/cache/conftool/dbconfig/20220425-035421-ladsgroup.json
* 03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26422 and previous config saved to /var/cache/conftool/dbconfig/20220425-033916-ladsgroup.json
* 03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26421 and previous config saved to /var/cache/conftool/dbconfig/20220425-032410-ladsgroup.json
* 03:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26420 and previous config saved to /var/cache/conftool/dbconfig/20220425-031959-ladsgroup.json
* 03:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26419 and previous config saved to /var/cache/conftool/dbconfig/20220425-031944-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26418 and previous config saved to /var/cache/conftool/dbconfig/20220425-030439-ladsgroup.json
* 02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26417 and previous config saved to /var/cache/conftool/dbconfig/20220425-024934-ladsgroup.json
* 02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26416 and previous config saved to /var/cache/conftool/dbconfig/20220425-023429-ladsgroup.json
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26415 and previous config saved to /var/cache/conftool/dbconfig/20220425-023020-ladsgroup.json
* 02:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26414 and previous config saved to /var/cache/conftool/dbconfig/20220425-023012-ladsgroup.json
* 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26413 and previous config saved to /var/cache/conftool/dbconfig/20220425-021507-ladsgroup.json
* 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26412 and previous config saved to /var/cache/conftool/dbconfig/20220425-020002-ladsgroup.json
* 01:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26411 and previous config saved to /var/cache/conftool/dbconfig/20220425-014457-ladsgroup.json
* 01:40 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: update new codfw1dev host (duration: 00m 54s)
* 01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
* 01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26410 and previous config saved to /var/cache/conftool/dbconfig/20220425-010952-ladsgroup.json
* 01:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26409 and previous config saved to /var/cache/conftool/dbconfig/20220425-010938-ladsgroup.json
* 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26408 and previous config saved to /var/cache/conftool/dbconfig/20220425-005432-ladsgroup.json
* 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26407 and previous config saved to /var/cache/conftool/dbconfig/20220425-003927-ladsgroup.json
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26406 and previous config saved to /var/cache/conftool/dbconfig/20220425-002422-ladsgroup.json
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26405 and previous config saved to /var/cache/conftool/dbconfig/20220425-001152-ladsgroup.json
* 00:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 00:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26404 and previous config saved to /var/cache/conftool/dbconfig/20220425-001144-ladsgroup.json


== 2022-04-24 ==
== 2022-08-07 ==
* 23:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26403 and previous config saved to /var/cache/conftool/dbconfig/20220424-235639-ladsgroup.json
* 19:58 taavi: taavi@mwmaint1002 ~ $ echo "https://upload.wikimedia.org/wikipedia/commons/1/15/Keep_tidy_ask.svg" {{!}} mwscript purgeList.php --wiki enwiki # [[phab:T314712|T314712]]
* 23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26402 and previous config saved to /var/cache/conftool/dbconfig/20220424-234134-ladsgroup.json
* 13:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32305 and previous config saved to /var/cache/conftool/dbconfig/20220807-135204-ladsgroup.json
* 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26401 and previous config saved to /var/cache/conftool/dbconfig/20220424-232629-ladsgroup.json
* 13:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26400 and previous config saved to /var/cache/conftool/dbconfig/20220424-232219-ladsgroup.json
* 13:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32304 and previous config saved to /var/cache/conftool/dbconfig/20220807-135143-ladsgroup.json
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P32303 and previous config saved to /var/cache/conftool/dbconfig/20220807-133637-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26399 and previous config saved to /var/cache/conftool/dbconfig/20220424-232205-ladsgroup.json
* 13:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P32302 and previous config saved to /var/cache/conftool/dbconfig/20220807-132131-ladsgroup.json
* 23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26398 and previous config saved to /var/cache/conftool/dbconfig/20220424-230700-ladsgroup.json
* 13:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32301 and previous config saved to /var/cache/conftool/dbconfig/20220807-130625-ladsgroup.json
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26397 and previous config saved to /var/cache/conftool/dbconfig/20220424-230136-ladsgroup.json
* 12:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32300 and previous config saved to /var/cache/conftool/dbconfig/20220807-120610-ladsgroup.json
* 22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26396 and previous config saved to /var/cache/conftool/dbconfig/20220424-225155-ladsgroup.json
* 12:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26395 and previous config saved to /var/cache/conftool/dbconfig/20220424-224631-ladsgroup.json
* 12:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26394 and previous config saved to /var/cache/conftool/dbconfig/20220424-223650-ladsgroup.json
* 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32299 and previous config saved to /var/cache/conftool/dbconfig/20220807-120549-ladsgroup.json
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26393 and previous config saved to /var/cache/conftool/dbconfig/20220424-223240-ladsgroup.json
* 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P32298 and previous config saved to /var/cache/conftool/dbconfig/20220807-115043-ladsgroup.json
* 22:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P32297 and previous config saved to /var/cache/conftool/dbconfig/20220807-113537-ladsgroup.json
* 22:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 11:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32296 and previous config saved to /var/cache/conftool/dbconfig/20220807-112031-ladsgroup.json
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26392 and previous config saved to /var/cache/conftool/dbconfig/20220424-223232-ladsgroup.json
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26391 and previous config saved to /var/cache/conftool/dbconfig/20220424-223126-ladsgroup.json
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26390 and previous config saved to /var/cache/conftool/dbconfig/20220424-221727-ladsgroup.json
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26389 and previous config saved to /var/cache/conftool/dbconfig/20220424-221621-ladsgroup.json
* 22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26388 and previous config saved to /var/cache/conftool/dbconfig/20220424-220222-ladsgroup.json
* 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26387 and previous config saved to /var/cache/conftool/dbconfig/20220424-214717-ladsgroup.json
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26386 and previous config saved to /var/cache/conftool/dbconfig/20220424-213440-ladsgroup.json
* 21:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26385 and previous config saved to /var/cache/conftool/dbconfig/20220424-213425-ladsgroup.json
* 21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26384 and previous config saved to /var/cache/conftool/dbconfig/20220424-211920-ladsgroup.json
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26383 and previous config saved to /var/cache/conftool/dbconfig/20220424-211607-ladsgroup.json
* 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26382 and previous config saved to /var/cache/conftool/dbconfig/20220424-211559-ladsgroup.json
* 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26381 and previous config saved to /var/cache/conftool/dbconfig/20220424-210521-ladsgroup.json
* 21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26380 and previous config saved to /var/cache/conftool/dbconfig/20220424-210415-ladsgroup.json
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26379 and previous config saved to /var/cache/conftool/dbconfig/20220424-210052-ladsgroup.json
* 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26378 and previous config saved to /var/cache/conftool/dbconfig/20220424-205016-ladsgroup.json
* 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26377 and previous config saved to /var/cache/conftool/dbconfig/20220424-204910-ladsgroup.json
* 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26376 and previous config saved to /var/cache/conftool/dbconfig/20220424-204547-ladsgroup.json
* 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26375 and previous config saved to /var/cache/conftool/dbconfig/20220424-203639-ladsgroup.json
* 20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26374 and previous config saved to /var/cache/conftool/dbconfig/20220424-203630-ladsgroup.json
* 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26373 and previous config saved to /var/cache/conftool/dbconfig/20220424-203511-ladsgroup.json
* 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26372 and previous config saved to /var/cache/conftool/dbconfig/20220424-203042-ladsgroup.json
* 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26371 and previous config saved to /var/cache/conftool/dbconfig/20220424-202125-ladsgroup.json
* 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26370 and previous config saved to /var/cache/conftool/dbconfig/20220424-202006-ladsgroup.json
* 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26369 and previous config saved to /var/cache/conftool/dbconfig/20220424-200620-ladsgroup.json
* 19:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26368 and previous config saved to /var/cache/conftool/dbconfig/20220424-195115-ladsgroup.json
* 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26367 and previous config saved to /var/cache/conftool/dbconfig/20220424-194705-ladsgroup.json
* 19:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26366 and previous config saved to /var/cache/conftool/dbconfig/20220424-194651-ladsgroup.json
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26365 and previous config saved to /var/cache/conftool/dbconfig/20220424-193635-ladsgroup.json
* 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26364 and previous config saved to /var/cache/conftool/dbconfig/20220424-193611-ladsgroup.json
* 19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26363 and previous config saved to /var/cache/conftool/dbconfig/20220424-193146-ladsgroup.json
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26362 and previous config saved to /var/cache/conftool/dbconfig/20220424-193028-ladsgroup.json
* 19:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 19:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26361 and previous config saved to /var/cache/conftool/dbconfig/20220424-193020-ladsgroup.json
* 19:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26360 and previous config saved to /var/cache/conftool/dbconfig/20220424-192106-ladsgroup.json
* 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26359 and previous config saved to /var/cache/conftool/dbconfig/20220424-191641-ladsgroup.json
* 19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26358 and previous config saved to /var/cache/conftool/dbconfig/20220424-191515-ladsgroup.json
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26357 and previous config saved to /var/cache/conftool/dbconfig/20220424-190601-ladsgroup.json
* 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26356 and previous config saved to /var/cache/conftool/dbconfig/20220424-190135-ladsgroup.json
* 19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26355 and previous config saved to /var/cache/conftool/dbconfig/20220424-190008-ladsgroup.json
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26354 and previous config saved to /var/cache/conftool/dbconfig/20220424-185724-ladsgroup.json
* 18:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26353 and previous config saved to /var/cache/conftool/dbconfig/20220424-185717-ladsgroup.json
* 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26352 and previous config saved to /var/cache/conftool/dbconfig/20220424-185056-ladsgroup.json
* 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26351 and previous config saved to /var/cache/conftool/dbconfig/20220424-184503-ladsgroup.json
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26350 and previous config saved to /var/cache/conftool/dbconfig/20220424-184212-ladsgroup.json
* 18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26349 and previous config saved to /var/cache/conftool/dbconfig/20220424-183813-ladsgroup.json
* 18:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26348 and previous config saved to /var/cache/conftool/dbconfig/20220424-183805-ladsgroup.json
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26347 and previous config saved to /var/cache/conftool/dbconfig/20220424-182707-ladsgroup.json
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26346 and previous config saved to /var/cache/conftool/dbconfig/20220424-182300-ladsgroup.json
* 18:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26345 and previous config saved to /var/cache/conftool/dbconfig/20220424-181201-ladsgroup.json
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26344 and previous config saved to /var/cache/conftool/dbconfig/20220424-180755-ladsgroup.json
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26343 and previous config saved to /var/cache/conftool/dbconfig/20220424-180555-ladsgroup.json
* 18:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26342 and previous config saved to /var/cache/conftool/dbconfig/20220424-180530-ladsgroup.json
* 18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26341 and previous config saved to /var/cache/conftool/dbconfig/20220424-180013-ladsgroup.json
* 18:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26340 and previous config saved to /var/cache/conftool/dbconfig/20220424-175250-ladsgroup.json
* 17:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26339 and previous config saved to /var/cache/conftool/dbconfig/20220424-175025-ladsgroup.json
* 17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26338 and previous config saved to /var/cache/conftool/dbconfig/20220424-174553-ladsgroup.json
* 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26337 and previous config saved to /var/cache/conftool/dbconfig/20220424-173955-ladsgroup.json
* 17:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26336 and previous config saved to /var/cache/conftool/dbconfig/20220424-173520-ladsgroup.json
* 17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26335 and previous config saved to /var/cache/conftool/dbconfig/20220424-172450-ladsgroup.json
* 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26334 and previous config saved to /var/cache/conftool/dbconfig/20220424-172015-ladsgroup.json
* 17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26333 and previous config saved to /var/cache/conftool/dbconfig/20220424-170945-ladsgroup.json
* 16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26332 and previous config saved to /var/cache/conftool/dbconfig/20220424-165439-ladsgroup.json
* 16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26331 and previous config saved to /var/cache/conftool/dbconfig/20220424-164748-ladsgroup.json
* 16:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 16:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26330 and previous config saved to /var/cache/conftool/dbconfig/20220424-163151-ladsgroup.json
* 16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 16:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 03:20 Amir1: optimizing flaggedtemplates on plwiki (s2) in db2088
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26329 and previous config saved to /var/cache/conftool/dbconfig/20220424-012830-ladsgroup.json
* 01:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26328 and previous config saved to /var/cache/conftool/dbconfig/20220424-011325-ladsgroup.json
* 00:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26327 and previous config saved to /var/cache/conftool/dbconfig/20220424-005820-ladsgroup.json
* 00:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26326 and previous config saved to /var/cache/conftool/dbconfig/20220424-004315-ladsgroup.json


== 2022-04-23 ==
== 2022-08-06 ==
* 23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26325 and previous config saved to /var/cache/conftool/dbconfig/20220423-232748-ladsgroup.json
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P32295 and previous config saved to /var/cache/conftool/dbconfig/20220806-175916-ladsgroup.json
* 23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 17:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 17:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 03:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 03:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26324 and previous config saved to /var/cache/conftool/dbconfig/20220423-232735-ladsgroup.json
* 03:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P26323 and previous config saved to /var/cache/conftool/dbconfig/20220423-231230-ladsgroup.json
* 03:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P26322 and previous config saved to /var/cache/conftool/dbconfig/20220423-225725-ladsgroup.json
* 03:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26321 and previous config saved to /var/cache/conftool/dbconfig/20220423-224220-ladsgroup.json
* 03:02 krinkle@deploy1002: Synchronized w/: {{Gerrit|I9067d47fab0324}} (duration: 03m 25s)
* 21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26320 and previous config saved to /var/cache/conftool/dbconfig/20220423-212332-ladsgroup.json
* 03:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 03:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 03:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26319 and previous config saved to /var/cache/conftool/dbconfig/20220423-212324-ladsgroup.json
* 02:41 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P26318 and previous config saved to /var/cache/conftool/dbconfig/20220423-210819-ladsgroup.json
* 02:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P26317 and previous config saved to /var/cache/conftool/dbconfig/20220423-205313-ladsgroup.json
* 02:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26316 and previous config saved to /var/cache/conftool/dbconfig/20220423-203808-ladsgroup.json
* 02:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26315 and previous config saved to /var/cache/conftool/dbconfig/20220423-191224-ladsgroup.json
* 02:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 19:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 02:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 19:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 02:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26314 and previous config saved to /var/cache/conftool/dbconfig/20220423-191216-ladsgroup.json
* 02:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P26313 and previous config saved to /var/cache/conftool/dbconfig/20220423-185711-ladsgroup.json
* 02:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P26312 and previous config saved to /var/cache/conftool/dbconfig/20220423-184206-ladsgroup.json
* 02:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26311 and previous config saved to /var/cache/conftool/dbconfig/20220423-182701-ladsgroup.json
* 16:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26310 and previous config saved to /var/cache/conftool/dbconfig/20220423-165939-ladsgroup.json
* 16:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 16:25 akosiaris: increase mw1335 and mw1336 weights on the jobrunner cluster from 1 to 4 (they were at %25 CPU usage). That should direct more traffic to them and lighten the load on the rest.
* 16:24 akosiaris@cumin1001: conftool action : set/weight=4; selector: cluster=jobrunner,name=mw1336.eqiad.wmnet
* 16:24 akosiaris@cumin1001: conftool action : set/weight=4; selector: cluster=jobrunner,name=mw1335.eqiad.wmnet
* 16:22 akosiaris@cumin1001: conftool action : set/weight=8; selector: cluster=jobrunner,name=mw1335.eqiad.wmnet
* 16:22 akosiaris@cumin1001: conftool action : set/weight=8; selector: cluster=jobrunner,name=mw1336.eqiad.wmnet
* 16:18 akosiaris: depool the videoscalers from the jobrunner cluster. Effectively split the 2 clusters that way. This should isolate the rest of the jobs from the video transcoding jobs reducing the latency that they are experiencing
* 16:17 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1446.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1445.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1440.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1439.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1438.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1437.eqiad.wmnet
* 16:16 akosiaris@cumin1001: conftool action : set/pooled=no; selector: cluster=jobrunner,name=mw1338.eqiad.wmnet
* 15:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 15:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 14:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26309 and previous config saved to /var/cache/conftool/dbconfig/20220423-142129-ladsgroup.json
* 14:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26308 and previous config saved to /var/cache/conftool/dbconfig/20220423-140624-ladsgroup.json
* 13:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P26307 and previous config saved to /var/cache/conftool/dbconfig/20220423-135119-ladsgroup.json
* 13:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26306 and previous config saved to /var/cache/conftool/dbconfig/20220423-133614-ladsgroup.json
* 12:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26305 and previous config saved to /var/cache/conftool/dbconfig/20220423-123558-ladsgroup.json
* 12:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 12:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 12:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26304 and previous config saved to /var/cache/conftool/dbconfig/20220423-123550-ladsgroup.json
* 12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P26303 and previous config saved to /var/cache/conftool/dbconfig/20220423-122045-ladsgroup.json
* 12:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P26302 and previous config saved to /var/cache/conftool/dbconfig/20220423-120540-ladsgroup.json
* 11:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26301 and previous config saved to /var/cache/conftool/dbconfig/20220423-115035-ladsgroup.json
* 11:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26300 and previous config saved to /var/cache/conftool/dbconfig/20220423-110511-ladsgroup.json
* 10:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26299 and previous config saved to /var/cache/conftool/dbconfig/20220423-105005-ladsgroup.json
* 10:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26298 and previous config saved to /var/cache/conftool/dbconfig/20220423-103500-ladsgroup.json
* 10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26297 and previous config saved to /var/cache/conftool/dbconfig/20220423-103135-ladsgroup.json
* 10:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 10:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 10:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26296 and previous config saved to /var/cache/conftool/dbconfig/20220423-103127-ladsgroup.json
* 10:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26295 and previous config saved to /var/cache/conftool/dbconfig/20220423-101955-ladsgroup.json
* 10:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P26294 and previous config saved to /var/cache/conftool/dbconfig/20220423-101622-ladsgroup.json
* 10:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P26293 and previous config saved to /var/cache/conftool/dbconfig/20220423-100115-ladsgroup.json
* 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26292 and previous config saved to /var/cache/conftool/dbconfig/20220423-094610-ladsgroup.json
* 09:38 elukey: `apt-get clean` on an-airflow1001 to free some space
* 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26291 and previous config saved to /var/cache/conftool/dbconfig/20220423-093443-ladsgroup.json
* 09:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 09:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 09:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26290 and previous config saved to /var/cache/conftool/dbconfig/20220423-093435-ladsgroup.json
* 09:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26289 and previous config saved to /var/cache/conftool/dbconfig/20220423-091930-ladsgroup.json
* 09:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26288 and previous config saved to /var/cache/conftool/dbconfig/20220423-090425-ladsgroup.json
* 08:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26287 and previous config saved to /var/cache/conftool/dbconfig/20220423-084920-ladsgroup.json
* 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26286 and previous config saved to /var/cache/conftool/dbconfig/20220423-083545-ladsgroup.json
* 08:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 08:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26285 and previous config saved to /var/cache/conftool/dbconfig/20220423-083532-ladsgroup.json
* 08:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26284 and previous config saved to /var/cache/conftool/dbconfig/20220423-082735-ladsgroup.json
* 08:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 08:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 08:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26283 and previous config saved to /var/cache/conftool/dbconfig/20220423-082726-ladsgroup.json
* 08:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26282 and previous config saved to /var/cache/conftool/dbconfig/20220423-082027-ladsgroup.json
* 08:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26281 and previous config saved to /var/cache/conftool/dbconfig/20220423-081221-ladsgroup.json
* 08:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26280 and previous config saved to /var/cache/conftool/dbconfig/20220423-080522-ladsgroup.json
* 07:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P26279 and previous config saved to /var/cache/conftool/dbconfig/20220423-075716-ladsgroup.json
* 07:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26278 and previous config saved to /var/cache/conftool/dbconfig/20220423-075017-ladsgroup.json
* 07:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26277 and previous config saved to /var/cache/conftool/dbconfig/20220423-074211-ladsgroup.json
* 07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26276 and previous config saved to /var/cache/conftool/dbconfig/20220423-073656-ladsgroup.json
* 07:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 07:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26275 and previous config saved to /var/cache/conftool/dbconfig/20220423-073648-ladsgroup.json
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26274 and previous config saved to /var/cache/conftool/dbconfig/20220423-072143-ladsgroup.json
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26273 and previous config saved to /var/cache/conftool/dbconfig/20220423-070638-ladsgroup.json
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26272 and previous config saved to /var/cache/conftool/dbconfig/20220423-065133-ladsgroup.json
* 06:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26271 and previous config saved to /var/cache/conftool/dbconfig/20220423-062503-ladsgroup.json
* 06:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 06:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 06:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26270 and previous config saved to /var/cache/conftool/dbconfig/20220423-062455-ladsgroup.json
* 06:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P26269 and previous config saved to /var/cache/conftool/dbconfig/20220423-060950-ladsgroup.json
* 05:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P26268 and previous config saved to /var/cache/conftool/dbconfig/20220423-055445-ladsgroup.json
* 05:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26267 and previous config saved to /var/cache/conftool/dbconfig/20220423-055118-ladsgroup.json
* 05:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 05:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26266 and previous config saved to /var/cache/conftool/dbconfig/20220423-053940-ladsgroup.json
* 05:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 05:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 05:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26265 and previous config saved to /var/cache/conftool/dbconfig/20220423-051219-ladsgroup.json
* 04:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26264 and previous config saved to /var/cache/conftool/dbconfig/20220423-045714-ladsgroup.json
* 04:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26263 and previous config saved to /var/cache/conftool/dbconfig/20220423-044209-ladsgroup.json
* 04:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26262 and previous config saved to /var/cache/conftool/dbconfig/20220423-042704-ladsgroup.json
* 04:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26261 and previous config saved to /var/cache/conftool/dbconfig/20220423-042001-ladsgroup.json
* 04:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 04:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 04:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26260 and previous config saved to /var/cache/conftool/dbconfig/20220423-041953-ladsgroup.json
* 04:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P26259 and previous config saved to /var/cache/conftool/dbconfig/20220423-040448-ladsgroup.json
* 03:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P26258 and previous config saved to /var/cache/conftool/dbconfig/20220423-034943-ladsgroup.json
* 03:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26257 and previous config saved to /var/cache/conftool/dbconfig/20220423-034558-ladsgroup.json
* 03:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 03:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 03:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26256 and previous config saved to /var/cache/conftool/dbconfig/20220423-034550-ladsgroup.json
* 03:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26255 and previous config saved to /var/cache/conftool/dbconfig/20220423-033438-ladsgroup.json
* 03:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26254 and previous config saved to /var/cache/conftool/dbconfig/20220423-033045-ladsgroup.json
* 03:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26253 and previous config saved to /var/cache/conftool/dbconfig/20220423-031540-ladsgroup.json
* 03:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26252 and previous config saved to /var/cache/conftool/dbconfig/20220423-030035-ladsgroup.json
* 02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26251 and previous config saved to /var/cache/conftool/dbconfig/20220423-021851-ladsgroup.json
* 02:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26250 and previous config saved to /var/cache/conftool/dbconfig/20220423-021826-ladsgroup.json
* 02:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 02:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 02:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26249 and previous config saved to /var/cache/conftool/dbconfig/20220423-021211-ladsgroup.json
* 02:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 02:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 02:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26248 and previous config saved to /var/cache/conftool/dbconfig/20220423-020346-ladsgroup.json
* 01:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26247 and previous config saved to /var/cache/conftool/dbconfig/20220423-014841-ladsgroup.json
* 01:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26246 and previous config saved to /var/cache/conftool/dbconfig/20220423-013450-ladsgroup.json
* 01:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 01:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 01:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26245 and previous config saved to /var/cache/conftool/dbconfig/20220423-013336-ladsgroup.json
* 01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26244 and previous config saved to /var/cache/conftool/dbconfig/20220423-011945-ladsgroup.json
* 01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26243 and previous config saved to /var/cache/conftool/dbconfig/20220423-010440-ladsgroup.json
* 00:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 00:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 00:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 00:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 00:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 00:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 00:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26242 and previous config saved to /var/cache/conftool/dbconfig/20220423-005613-ladsgroup.json
* 00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26241 and previous config saved to /var/cache/conftool/dbconfig/20220423-004935-ladsgroup.json
* 00:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26240 and previous config saved to /var/cache/conftool/dbconfig/20220423-004617-ladsgroup.json
* 00:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 00:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 00:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26239 and previous config saved to /var/cache/conftool/dbconfig/20220423-004108-ladsgroup.json
* 00:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26238 and previous config saved to /var/cache/conftool/dbconfig/20220423-002603-ladsgroup.json
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26237 and previous config saved to /var/cache/conftool/dbconfig/20220423-002352-ladsgroup.json
* 00:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 00:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26236 and previous config saved to /var/cache/conftool/dbconfig/20220423-002344-ladsgroup.json
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26235 and previous config saved to /var/cache/conftool/dbconfig/20220423-001058-ladsgroup.json
* 00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26234 and previous config saved to /var/cache/conftool/dbconfig/20220423-000839-ladsgroup.json


== 2022-04-22 ==
== 2022-08-05 ==
* 23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26233 and previous config saved to /var/cache/conftool/dbconfig/20220422-235334-ladsgroup.json
* 22:20 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@71fe016]: Fix schedule_interval for image_recommendation_weekly (duration: 02m 01s)
* 23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26232 and previous config saved to /var/cache/conftool/dbconfig/20220422-233829-ladsgroup.json
* 22:18 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@71fe016]: Fix schedule_interval for image_recommendation_weekly
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26231 and previous config saved to /var/cache/conftool/dbconfig/20220422-232210-ladsgroup.json
* 17:08 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1195.eqiad.wmnet with OS bullseye
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 16:54 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1194.eqiad.wmnet with OS bullseye
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 16:53 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage
* 23:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26230 and previous config saved to /var/cache/conftool/dbconfig/20220422-232147-ladsgroup.json
* 16:49 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1195.eqiad.wmnet with reason: host reimage
* 23:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26229 and previous config saved to /var/cache/conftool/dbconfig/20220422-230642-ladsgroup.json
* 16:41 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
* 22:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26228 and previous config saved to /var/cache/conftool/dbconfig/20220422-225735-ladsgroup.json
* 16:37 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
* 22:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 16:34 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1195.eqiad.wmnet with OS bullseye
* 22:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 16:27 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp203[56]\.codfw\.wmnet,service=varnish-fe
* 22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26227 and previous config saved to /var/cache/conftool/dbconfig/20220422-225136-ladsgroup.json
* 16:27 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp203[56]\.codfw\.wmnet,service=ats-be
* 22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26226 and previous config saved to /var/cache/conftool/dbconfig/20220422-223631-ladsgroup.json
* 16:27 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp203[56]\.codfw\.wmnet,service=ats-tls
* 22:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:26 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1194.eqiad.wmnet with OS bullseye
* 22:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:25 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1193.eqiad.wmnet with OS bullseye
* 22:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26225 and previous config saved to /var/cache/conftool/dbconfig/20220422-222203-ladsgroup.json
* 16:21 pt1979@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host db1192.eqiad.wmnet with OS bullseye
* 22:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26224 and previous config saved to /var/cache/conftool/dbconfig/20220422-220658-ladsgroup.json
* 16:12 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@8489923]: [[phab:T304954|T304954]]: Automate imagesuggestion imports (duration: 02m 03s)
* 21:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26223 and previous config saved to /var/cache/conftool/dbconfig/20220422-215153-ladsgroup.json
* 16:11 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage
* 21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26222 and previous config saved to /var/cache/conftool/dbconfig/20220422-213648-ladsgroup.json
* 16:11 milimetric@deploy1002: Finished deploy [analytics/refinery@fe7bf9e]: Hotfix for webrequest load refine, now with FORCE :) (duration: 06m 09s)
* 21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26221 and previous config saved to /var/cache/conftool/dbconfig/20220422-213617-ladsgroup.json
* 16:10 dcausse@deploy1002: Started deploy [wikimedia/discovery/analytics@8489923]: [[phab:T304954|T304954]]: Automate imagesuggestion imports
* 21:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:07 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1193.eqiad.wmnet with reason: host reimage
* 21:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:07 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage
* 21:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26220 and previous config saved to /var/cache/conftool/dbconfig/20220422-213609-ladsgroup.json
* 16:05 milimetric@deploy1002: Started deploy [analytics/refinery@fe7bf9e]: Hotfix for webrequest load refine, now with FORCE :)
* 21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26219 and previous config saved to /var/cache/conftool/dbconfig/20220422-212104-ladsgroup.json
* 16:04 milimetric@deploy1002: Finished deploy [analytics/refinery@fe7bf9e]: Hotfix for webrequest load refine (duration: 34m 38s)
* 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26218 and previous config saved to /var/cache/conftool/dbconfig/20220422-210559-ladsgroup.json
* 16:03 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1192.eqiad.wmnet with reason: host reimage
* 20:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26217 and previous config saved to /var/cache/conftool/dbconfig/20220422-205538-ladsgroup.json
* 15:55 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1193.eqiad.wmnet with OS bullseye
* 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26216 and previous config saved to /var/cache/conftool/dbconfig/20220422-205053-ladsgroup.json
* 15:52 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1191.eqiad.wmnet with OS bullseye
* 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26215 and previous config saved to /var/cache/conftool/dbconfig/20220422-204547-ladsgroup.json
* 15:51 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1192.eqiad.wmnet with OS bullseye
* 20:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 15:42 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1190.eqiad.wmnet with OS bullseye
* 20:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 15:38 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
* 20:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26214 and previous config saved to /var/cache/conftool/dbconfig/20220422-204033-ladsgroup.json
* 15:34 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
* 20:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26213 and previous config saved to /var/cache/conftool/dbconfig/20220422-202903-ladsgroup.json
* 15:30 milimetric@deploy1002: Started deploy [analytics/refinery@fe7bf9e]: Hotfix for webrequest load refine
* 20:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 15:28 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1190.eqiad.wmnet with reason: host reimage
* 20:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 15:25 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1190.eqiad.wmnet with reason: host reimage
* 20:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 15:24 jbond: upload trapperkeeper-metrics-clojure to puppet7 component
* 20:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 15:22 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1191.eqiad.wmnet with OS bullseye
* 20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P26212 and previous config saved to /var/cache/conftool/dbconfig/20220422-202528-ladsgroup.json
* 15:19 jbond: upload puppetlabs-http-client-clojur to puppet7 component
* 20:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26211 and previous config saved to /var/cache/conftool/dbconfig/20220422-201023-ladsgroup.json
* 15:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26210 and previous config saved to /var/cache/conftool/dbconfig/20220422-200605-ladsgroup.json
* 15:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 15:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 15:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 15:14 dancy@deploy1002: Finished scap: Backport for [[gerrit:820653]] scap gitignore: ignore all files under the `scap` directory (duration: 04m 41s)
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 15:11 jbond: upload jolokia to puppet7 component
* 20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 15:10 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1185.eqiad.wmnet with OS bullseye
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 15:09 dancy@deploy1002: Started scap: Backport for [[gerrit:820653]] scap gitignore: ignore all files under the `scap` directory
* 20:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 15:09 jbond: upload test-chuck-clojure to puppet7 component
* 20:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 15:05 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1190.eqiad.wmnet with OS bullseye
* 20:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 15:04 jbond: upload test-check-clojure to puppet7 component
* 20:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 14:57 jbond: upload nippy-clojure to puppet7 component
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 14:56 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 14:52 pt1979@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 14:43 jbond: upload fressian to puppet7 component
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 14:40 pt1979@cumin1001: START - Cookbook sre.hosts.reimage for host db1185.eqiad.wmnet with OS bullseye
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 14:40 jbond: upload test-generative-clojure to puppet7 component
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 14:35 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26209 and previous config saved to /var/cache/conftool/dbconfig/20220422-191935-ladsgroup.json
* 14:34 jbond: upload data-generators-clojure to puppet7 component
* 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26208 and previous config saved to /var/cache/conftool/dbconfig/20220422-191820-ladsgroup.json
* 14:31 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 19:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 14:23 jbond: upload encore-clojure to puppet7 component
* 19:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 14:17 jbond: upload truss-clojure to puppet7 component
* 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26207 and previous config saved to /var/cache/conftool/dbconfig/20220422-191812-ladsgroup.json
* 14:13 jbond: upload structured-logging-clojure to puppet7 component
* 19:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 14:06 jbond: upload murphy-clojure to puppet7 component
* 19:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 13:57 jbond: upload logstash-logback-encoder-7.2 to puppet7 component
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26206 and previous config saved to /var/cache/conftool/dbconfig/20220422-190632-ladsgroup.json
* 13:49 jbond: upload kitchensink-clojure to puppet7 component
* 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26205 and previous config saved to /var/cache/conftool/dbconfig/20220422-190306-ladsgroup.json
* 13:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool hosts with fragile power supply ([[phab:T314559|T314559]] [[phab:T314628|T314628]])', diff saved to https://phabricator.wikimedia.org/P32292 and previous config saved to /var/cache/conftool/dbconfig/20220805-132709-ladsgroup.json
* 18:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26204 and previous config saved to /var/cache/conftool/dbconfig/20220422-185126-ladsgroup.json
* 13:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
* 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P26203 and previous config saved to /var/cache/conftool/dbconfig/20220422-184801-ladsgroup.json
* 13:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on db2095.codfw.wmnet with reason: Maintenance
* 18:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26202 and previous config saved to /var/cache/conftool/dbconfig/20220422-183621-ladsgroup.json
* 13:09 sukhe: repool codfw
* 18:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26201 and previous config saved to /var/cache/conftool/dbconfig/20220422-183256-ladsgroup.json
* 13:02 jbond: upload honeysql-clojure to puppet7 component
* 18:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26200 and previous config saved to /var/cache/conftool/dbconfig/20220422-182116-ladsgroup.json
* 12:53 _joe_: progressive repool of services in codfw
* 17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26199 and previous config saved to /var/cache/conftool/dbconfig/20220422-173242-ladsgroup.json
* 12:24 moritzm: installing nano bugfix updates from bullseye point release
* 17:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 11:50 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: sync
* 17:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 11:40 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: sync
* 17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26198 and previous config saved to /var/cache/conftool/dbconfig/20220422-173234-ladsgroup.json
* 11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repool after PDU maint on D3 ([[phab:T310146|T310146]])', diff saved to https://phabricator.wikimedia.org/P32291 and previous config saved to /var/cache/conftool/dbconfig/20220805-113729-ladsgroup.json
* 17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26197 and previous config saved to /var/cache/conftool/dbconfig/20220422-173031-ladsgroup.json
* 11:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repool after PDU maint on C6 ([[phab:T310145|T310145]])', diff saved to https://phabricator.wikimedia.org/P32290 and previous config saved to /var/cache/conftool/dbconfig/20220805-113555-ladsgroup.json
* 17:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 11:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repool after PDU maint on C5 ([[phab:T310145|T310145]])', diff saved to https://phabricator.wikimedia.org/P32289 and previous config saved to /var/cache/conftool/dbconfig/20220805-113436-ladsgroup.json
* 17:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 10:46 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: sync
* 17:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26196 and previous config saved to /var/cache/conftool/dbconfig/20220422-173022-ladsgroup.json
* 10:36 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: sync
* 17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26195 and previous config saved to /var/cache/conftool/dbconfig/20220422-171727-ladsgroup.json
* 10:17 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: sync
* 17:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26194 and previous config saved to /var/cache/conftool/dbconfig/20220422-171517-ladsgroup.json
* 10:12 Amir1: dbmaint at s4@codfw ([[phab:T312863|T312863]])
* 17:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 10:07 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: sync
* 17:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 09:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 12 hosts with reason: Maintenance
* 17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 09:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 12 hosts with reason: Maintenance
* 17:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 09:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 17:04 krinkle@deploy1002: Synchronized static/: {{Gerrit|I5cf2340b3b0358}} (duration: 00m 58s)
* 09:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P26193 and previous config saved to /var/cache/conftool/dbconfig/20220422-170222-ladsgroup.json
* 00:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8 days, 0:00:00 on gerrit2001.wikimedia.org with reason: decom, replaced by gerrit2002
* 17:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26192 and previous config saved to /var/cache/conftool/dbconfig/20220422-170012-ladsgroup.json
* 00:53 dzahn@cumin1001: START - Cookbook sre.hosts.downtime for 8 days, 0:00:00 on gerrit2001.wikimedia.org with reason: decom, replaced by gerrit2002
* 16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26191 and previous config saved to /var/cache/conftool/dbconfig/20220422-164717-ladsgroup.json
* 00:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for gerrit2002.wikimedia.org
* 16:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26190 and previous config saved to /var/cache/conftool/dbconfig/20220422-164507-ladsgroup.json
* 00:53 dzahn@cumin1001: START - Cookbook sre.hosts.remove-downtime for gerrit2002.wikimedia.org
* 16:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26189 and previous config saved to /var/cache/conftool/dbconfig/20220422-164359-ladsgroup.json
* 00:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8 days, 0:00:00 on gerrit2002.wikimedia.org with reason: decom, replaced by gerrit2002
* 16:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 00:52 dzahn@cumin1001: START - Cookbook sre.hosts.downtime for 8 days, 0:00:00 on gerrit2002.wikimedia.org with reason: decom, replaced by gerrit2002
* 16:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 00:18 mutante: restarting gerrit for config change - removing old replica [[phab:T313250|T313250]]
* 16:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26188 and previous config saved to /var/cache/conftool/dbconfig/20220422-164350-ladsgroup.json
* 16:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26187 and previous config saved to /var/cache/conftool/dbconfig/20220422-162845-ladsgroup.json
* 16:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P26186 and previous config saved to /var/cache/conftool/dbconfig/20220422-161340-ladsgroup.json
* 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26185 and previous config saved to /var/cache/conftool/dbconfig/20220422-160342-ladsgroup.json
* 16:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 16:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 15:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26184 and previous config saved to /var/cache/conftool/dbconfig/20220422-155835-ladsgroup.json
* 15:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26183 and previous config saved to /var/cache/conftool/dbconfig/20220422-155617-ladsgroup.json
* 15:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 15:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 15:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26182 and previous config saved to /var/cache/conftool/dbconfig/20220422-155609-ladsgroup.json
* 15:42 Amir1: cleaning up all of old email tokens in s2
* 15:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26181 and previous config saved to /var/cache/conftool/dbconfig/20220422-154104-ladsgroup.json
* 15:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26180 and previous config saved to /var/cache/conftool/dbconfig/20220422-152559-ladsgroup.json
* 15:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 15:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 15:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26179 and previous config saved to /var/cache/conftool/dbconfig/20220422-152401-ladsgroup.json
* 15:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26178 and previous config saved to /var/cache/conftool/dbconfig/20220422-151053-ladsgroup.json
* 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26177 and previous config saved to /var/cache/conftool/dbconfig/20220422-150856-ladsgroup.json
* 15:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26176 and previous config saved to /var/cache/conftool/dbconfig/20220422-150836-ladsgroup.json
* 15:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 15:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 15:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 15:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26175 and previous config saved to /var/cache/conftool/dbconfig/20220422-145351-ladsgroup.json
* 14:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 14:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 14:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26174 and previous config saved to /var/cache/conftool/dbconfig/20220422-143846-ladsgroup.json
* 14:01 Amir1: removing all old user_email_token_expires rows in zhwiki
* 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26173 and previous config saved to /var/cache/conftool/dbconfig/20220422-135334-ladsgroup.json
* 13:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 13:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 13:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26172 and previous config saved to /var/cache/conftool/dbconfig/20220422-135326-ladsgroup.json
* 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26171 and previous config saved to /var/cache/conftool/dbconfig/20220422-133820-ladsgroup.json
* 13:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26170 and previous config saved to /var/cache/conftool/dbconfig/20220422-132315-ladsgroup.json
* 13:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 13:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 13:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 13:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 13:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26169 and previous config saved to /var/cache/conftool/dbconfig/20220422-130810-ladsgroup.json
* 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26168 and previous config saved to /var/cache/conftool/dbconfig/20220422-125447-ladsgroup.json
* 12:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 12:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26167 and previous config saved to /var/cache/conftool/dbconfig/20220422-125439-ladsgroup.json
* 12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26166 and previous config saved to /var/cache/conftool/dbconfig/20220422-123934-ladsgroup.json
* 12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26165 and previous config saved to /var/cache/conftool/dbconfig/20220422-122429-ladsgroup.json
* 12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26164 and previous config saved to /var/cache/conftool/dbconfig/20220422-120924-ladsgroup.json
* 11:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26163 and previous config saved to /var/cache/conftool/dbconfig/20220422-115626-ladsgroup.json
* 11:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 11:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 11:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 11:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26162 and previous config saved to /var/cache/conftool/dbconfig/20220422-115556-ladsgroup.json
* 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26161 and previous config saved to /var/cache/conftool/dbconfig/20220422-114051-ladsgroup.json
* 11:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26160 and previous config saved to /var/cache/conftool/dbconfig/20220422-112546-ladsgroup.json
* 11:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26159 and previous config saved to /var/cache/conftool/dbconfig/20220422-111041-ladsgroup.json
* 10:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 10:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 10:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 10:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 10:17 reedy@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/TimedMediaHandler/: [[phab:T306697|T306697]] (duration: 00m 50s)
* 10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26158 and previous config saved to /var/cache/conftool/dbconfig/20220422-101026-ladsgroup.json
* 10:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 10:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 10:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26157 and previous config saved to /var/cache/conftool/dbconfig/20220422-101018-ladsgroup.json
* 09:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26156 and previous config saved to /var/cache/conftool/dbconfig/20220422-095513-ladsgroup.json
* 09:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P26155 and previous config saved to /var/cache/conftool/dbconfig/20220422-094008-ladsgroup.json
* 09:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26154 and previous config saved to /var/cache/conftool/dbconfig/20220422-092503-ladsgroup.json
* 08:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26153 and previous config saved to /var/cache/conftool/dbconfig/20220422-084431-ladsgroup.json
* 08:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 08:44 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 08:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26152 and previous config saved to /var/cache/conftool/dbconfig/20220422-084418-ladsgroup.json
* 08:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26151 and previous config saved to /var/cache/conftool/dbconfig/20220422-082913-ladsgroup.json
* 08:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P26150 and previous config saved to /var/cache/conftool/dbconfig/20220422-081408-ladsgroup.json
* 07:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26149 and previous config saved to /var/cache/conftool/dbconfig/20220422-075903-ladsgroup.json
* 07:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1174 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26148 and previous config saved to /var/cache/conftool/dbconfig/20220422-074520-ladsgroup.json
* 07:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 07:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 07:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26147 and previous config saved to /var/cache/conftool/dbconfig/20220422-074512-ladsgroup.json
* 07:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26146 and previous config saved to /var/cache/conftool/dbconfig/20220422-073007-ladsgroup.json
* 07:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26145 and previous config saved to /var/cache/conftool/dbconfig/20220422-071502-ladsgroup.json
* 06:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26144 and previous config saved to /var/cache/conftool/dbconfig/20220422-065957-ladsgroup.json
* 06:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26143 and previous config saved to /var/cache/conftool/dbconfig/20220422-065332-ladsgroup.json
* 06:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26142 and previous config saved to /var/cache/conftool/dbconfig/20220422-063827-ladsgroup.json
* 06:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26141 and previous config saved to /var/cache/conftool/dbconfig/20220422-062322-ladsgroup.json
* 06:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26140 and previous config saved to /var/cache/conftool/dbconfig/20220422-061304-ladsgroup.json
* 06:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 06:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 06:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26139 and previous config saved to /var/cache/conftool/dbconfig/20220422-060816-ladsgroup.json
* 05:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 05:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 05:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26138 and previous config saved to /var/cache/conftool/dbconfig/20220422-053246-ladsgroup.json
* 05:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26137 and previous config saved to /var/cache/conftool/dbconfig/20220422-051740-ladsgroup.json
* 05:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26136 and previous config saved to /var/cache/conftool/dbconfig/20220422-050802-ladsgroup.json
* 05:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 05:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 05:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P26135 and previous config saved to /var/cache/conftool/dbconfig/20220422-050235-ladsgroup.json
* 04:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26134 and previous config saved to /var/cache/conftool/dbconfig/20220422-044730-ladsgroup.json
* 04:23 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 04:23 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 04:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26133 and previous config saved to /var/cache/conftool/dbconfig/20220422-040325-ladsgroup.json
* 04:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 04:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 04:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26132 and previous config saved to /var/cache/conftool/dbconfig/20220422-040316-ladsgroup.json
* 03:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26131 and previous config saved to /var/cache/conftool/dbconfig/20220422-034811-ladsgroup.json
* 03:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 03:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 03:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P26130 and previous config saved to /var/cache/conftool/dbconfig/20220422-033306-ladsgroup.json
* 03:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26129 and previous config saved to /var/cache/conftool/dbconfig/20220422-031801-ladsgroup.json
* 02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 02:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 02:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 02:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26127 and previous config saved to /var/cache/conftool/dbconfig/20220422-024512-ladsgroup.json
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26126 and previous config saved to /var/cache/conftool/dbconfig/20220422-023007-ladsgroup.json
* 02:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26125 and previous config saved to /var/cache/conftool/dbconfig/20220422-022544-ladsgroup.json
* 02:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 02:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
* 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26124 and previous config saved to /var/cache/conftool/dbconfig/20220422-021502-ladsgroup.json
* 01:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26123 and previous config saved to /var/cache/conftool/dbconfig/20220422-015957-ladsgroup.json
* 01:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 01:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 01:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance
* 01:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance
* 01:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 01:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance
* 01:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26122 and previous config saved to /var/cache/conftool/dbconfig/20220422-010645-ladsgroup.json
* 00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26121 and previous config saved to /var/cache/conftool/dbconfig/20220422-005942-ladsgroup.json
* 00:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 00:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 00:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26120 and previous config saved to /var/cache/conftool/dbconfig/20220422-005934-ladsgroup.json
* 00:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26119 and previous config saved to /var/cache/conftool/dbconfig/20220422-005140-ladsgroup.json
* 00:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26118 and previous config saved to /var/cache/conftool/dbconfig/20220422-004429-ladsgroup.json
* 00:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26117 and previous config saved to /var/cache/conftool/dbconfig/20220422-003634-ladsgroup.json
* 00:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26116 and previous config saved to /var/cache/conftool/dbconfig/20220422-002924-ladsgroup.json
* 00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26115 and previous config saved to /var/cache/conftool/dbconfig/20220422-002129-ladsgroup.json
* 00:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26114 and previous config saved to /var/cache/conftool/dbconfig/20220422-001418-ladsgroup.json
* 00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26113 and previous config saved to /var/cache/conftool/dbconfig/20220422-000732-ladsgroup.json
* 00:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 00:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1136.eqiad.wmnet with reason: Maintenance
* 00:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26112 and previous config saved to /var/cache/conftool/dbconfig/20220422-000708-ladsgroup.json


== 2022-04-21 ==
== 2022-08-04 ==
* 23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26111 and previous config saved to /var/cache/conftool/dbconfig/20220421-235814-ladsgroup.json
* 23:07 mutante: switching gerrit-replica.wikimedia.org to new machine gerrit2002, dropping gerrit-replica-new.wikimedia.org [[phab:T313250|T313250]]
* 23:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 21:07 ryankemper@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 23:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 20:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26110 and previous config saved to /var/cache/conftool/dbconfig/20220421-235203-ladsgroup.json
* 20:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26109 and previous config saved to /var/cache/conftool/dbconfig/20220421-233658-ladsgroup.json
* 20:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26108 and previous config saved to /var/cache/conftool/dbconfig/20220421-233212-ladsgroup.json
* 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 23:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 20:56 thcipriani@deploy1002: Finished
* 23:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 23:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26107 and previous config saved to /var/cache/conftool/dbconfig/20220421-232153-ladsgroup.json
* 23:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 23:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 23:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 23:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26106 and previous config saved to /var/cache/conftool/dbconfig/20220421-231913-ladsgroup.json
* 23:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26105 and previous config saved to /var/cache/conftool/dbconfig/20220421-231707-ladsgroup.json
* 23:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26104
* 16:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:28 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820376{{!}}Start reading from new templatelinks columns in commons (T306673)]] (duration: 03m 00s)
* 16:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:19 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 16:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:19 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 16:17 brett: deploying authdns - geodns: Map out African countries by DC latency ([[phab:T311472|T311472]])
* 16:19 kormat@cumin1001: dbctl commit (dc=all): 'db1158 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25775 and previous config saved to /var/cache/conftool/dbconfig/20220420-161914-kormat.json
* 16:12 cwhite: poweroff logstash2028 - [[phab:T310145|T310145]]
* 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25774 and previous config saved to /var/cache/conftool/dbconfig/20220420-161828-ladsgroup.json
* 16:06 Emperor: shutdown ms-be20[39,49,54].codfw.wmnet,thanos-be2003 for PDU swap [[phab:T310145|T310145]]
* 16:15 kormat@cumin1001: dbctl commit (dc=all): 'es1028 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25773 and previous config saved to /var/cache/conftool/dbconfig/20220420-161511-kormat.json
* 16:03 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be[2036,2049,2054].codfw.wmnet,thanos-be2003.codfw.wmnet with reason: PDU work
* 16:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 16:02 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be[2036,2049,2054].codfw.wmnet,thanos-be2003.codfw.wmnet with reason: PDU work
* 16:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1028.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 16:02 ebysans@deploy1002: Started deploy [analytics/refinery@2553288]: Regular analytics weekly train [analytics/refinery@2553288]
* 16:14 kormat@cumin1001: dbctl commit (dc=all): 'Change es3 'master' to es1031 [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25772 and previous config saved to /var/cache/conftool/dbconfig/20220420-161453-kormat.json
* 15:50 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic2048.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 16:13 kormat@cumin1001: dbctl commit (dc=all): 'db1158 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25771 and previous config saved to /var/cache/conftool/dbconfig/20220420-161353-kormat.json
* 15:50 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic2048.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 16:13 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1158.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:43 damilare: payments-wiki upgraded from {{Gerrit|0e4a5b3b}} to {{Gerrit|6880236d}}
* 16:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1158.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:37 _joe_: uncordoning ml-serve200<nowiki>{</nowiki>1,6<nowiki>}</nowiki>
* 16:13 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1158 [[phab:T303174|T303174]]
* 15:27 sukhe: power off cp2037,cp2038: PDU upgrade
* 16:13 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Rebooting db1158 [[phab:T303174|T303174]]
* 15:25 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:30:00 on phab2001.codfw.wmnet with reason: PDU swap
* 16:13 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1007.eqiad.wmnet with reason: host reimage
* 15:25 jelto: power off phab2001
* 16:12 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:25 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 3:30:00 on phab2001.codfw.wmnet with reason: PDU swap
* 16:12 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:25 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cp[2037-2038].codfw.wmnet with reason: shutdown for PDU upgrade
* 16:11 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25770 and previous config saved to /var/cache/conftool/dbconfig/20220420-161123-kormat.json
* 15:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on cp[2037-2038].codfw.wmnet with reason: shutdown for PDU upgrade
* 16:09 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1007.eqiad.wmnet with reason: host reimage
* 15:24 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp203[78]\.codfw\.wmnet,service=varnish-fe
* 16:09 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25769 and previous config saved to /var/cache/conftool/dbconfig/20220420-160926-kormat.json
* 15:23 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp203[78]\.codfw\.wmnet,service=ats-be
* 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25768 and previous config saved to /var/cache/conftool/dbconfig/20220420-160322-ladsgroup.json
* 15:23 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp203[78]\.codfw\.wmnet,service=ats-tls
* 15:57 hnowlan@deploy1002: Finished deploy [restbase/deploy@0205f1d]: Bump mediawiki-title to 0.7.5 (duration: 15m 35s)
* 15:21 XioNoX: un-drain codfw-ulsfo link - [[phab:T310310|T310310]]
* 15:56 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1007.eqiad.wmnet with OS bullseye
* 15:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2116,2127,2167-2168].codfw.wmnet,es2022.codfw.wmnet with reason: Maintenance ([[phab:T310145|T310145]])
* 15:56 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25767 and previous config saved to /var/cache/conftool/dbconfig/20220420-155619-kormat.json
* 15:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db[2116,2127,2167-2168].codfw.wmnet,es2022.codfw.wmnet with reason: Maintenance ([[phab:T310145|T310145]])
* 15:55 jynus@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1007.eqiad.wmnet with OS bullseye
* 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool C6 for PDU maint ([[phab:T310145|T310145]])', diff saved to https://phabricator.wikimedia.org/P32285 and previous config saved to /var/cache/conftool/dbconfig/20220804-151958-ladsgroup.json
* 15:54 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25766 and previous config saved to /var/cache/conftool/dbconfig/20220420-155422-kormat.json
* 15:16 btullis@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 15:53 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25765 and previous config saved to /var/cache/conftool/dbconfig/20220420-155318-kormat.json
* 15:16 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on restbase[2016,2020,2025].codfw.wmnet with reason: PDU maintenance
* 15:50 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25764 and previous config saved to /var/cache/conftool/dbconfig/20220420-155051-kormat.json
* 15:16 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 3:00:00 on restbase[2016,2020,2025].codfw.wmnet with reason: PDU maintenance
* 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25763 and previous config saved to /var/cache/conftool/dbconfig/20220420-154817-ladsgroup.json
* 15:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2114,2126,2166].codfw.wmnet with reason: Maintenance ([[phab:T310145|T310145]])
* 15:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25762 and previous config saved to /var/cache/conftool/dbconfig/20220420-154734-ladsgroup.json
* 15:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db[2114,2126,2166].codfw.wmnet with reason: Maintenance ([[phab:T310145|T310145]])
* 15:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 15:13 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp203[12]\.codfw\.wmnet,service=varnish-fe
* 15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 15:13 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp203[12]\.codfw\.wmnet,service=ats-be
* 15:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 15:13 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp203[12]\.codfw\.wmnet,service=ats-tls
* 15:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 15:12 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be[2058,2064].codfw.wmnet
* 15:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25761 and previous config saved to /var/cache/conftool/dbconfig/20220420-154635-ladsgroup.json
* 15:12 mvernon@cumin1001: START - Cookbook sre.hosts.remove-downtime for ms-be[2058,2064].codfw.wmnet
* 15:44 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25760 and previous config saved to /var/cache/conftool/dbconfig/20220420-154427-kormat.json
* 15:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool hosts for PDU maint ([[phab:T310145|T310145]])', diff saved to https://phabricator.wikimedia.org/P32284 and previous config saved to /var/cache/conftool/dbconfig/20220804-151121-ladsgroup.json
* 15:41 hnowlan@deploy1002: Started deploy [restbase/deploy@0205f1d]: Bump mediawiki-title to 0.7.5
* 15:09 godog: poweroff logstash2002 - [[phab:T310145|T310145]]
* 15:41 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25759 and previous config saved to /var/cache/conftool/dbconfig/20220420-154115-kormat.json
* 15:07 _joe_: pwoering down mc203<nowiki>{</nowiki>0,1<nowiki>}</nowiki>
* 15:39 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25758 and previous config saved to /var/cache/conftool/dbconfig/20220420-153918-kormat.json
* 15:07 root@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on logstash2002.codfw.wmnet with reason: pdu
* 15:38 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25757 and previous config saved to /var/cache/conftool/dbconfig/20220420-153814-kormat.json
* 15:06 root@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on logstash2002.codfw.wmnet with reason: pdu
* 15:35 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25756 and previous config saved to /var/cache/conftool/dbconfig/20220420-153547-kormat.json
* 15:05 btullis@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25755 and previous config saved to /var/cache/conftool/dbconfig/20220420-153312-ladsgroup.json
* 14:58 jelto: power off mc20[30-31]
* 15:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25754 and previous config saved to /var/cache/conftool/dbconfig/20220420-153130-ladsgroup.json
* 14:56 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mc[2030-2031].codfw.wmnet with reason: PDU swap
* 15:29 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25753 and previous config saved to /var/cache/conftool/dbconfig/20220420-152923-kormat.json
* 14:56 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on mc[2030-2031].codfw.wmnet with reason: PDU swap
* 15:26 kormat@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25752 and previous config saved to /var/cache/conftool/dbconfig/20220420-152611-kormat.json
* 14:56 XioNoX: draining codfw-ulsfo link - [[phab:T310310|T310310]]
* 15:24 kormat@cumin1001: dbctl commit (dc=all): 'es1033 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25751 and previous config saved to /var/cache/conftool/dbconfig/20220420-152414-kormat.json
* 14:36 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps2009.codfw.wmnet
* 15:23 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host backup1007.eqiad.wmnet with OS bullseye
* 14:35 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps2007.codfw.wmnet
* 15:23 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25750 and previous config saved to /var/cache/conftool/dbconfig/20220420-152310-kormat.json
* 14:35 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2025.codfw.wmnet
* 15:20 kormat@cumin1001: dbctl commit (dc=all): 'es1025 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25749 and previous config saved to /var/cache/conftool/dbconfig/20220420-152044-kormat.json
* 14:35 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2020.codfw.wmnet
* 15:20 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25748 and previous config saved to /var/cache/conftool/dbconfig/20220420-152043-kormat.json
* 14:35 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2016.codfw.wmnet
* 15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:32 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs2011.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 15:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:31 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs2011.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:25 jelto: power off gitlab-runner2003
* 15:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:25 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:30:00 on gitlab-runner2003.codfw.wmnet with reason: PDU swap
* 15:20 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:25 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs2001.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 15:20 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:24 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs2001.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 15:18 moritzm: installing wireshark security updates
* 14:24 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 4:30:00 on gitlab-runner2003.codfw.wmnet with reason: PDU swap
* 15:16 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
* 14:23 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic2032.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25747 and previous config saved to /var/cache/conftool/dbconfig/20220420-151625-ladsgroup.json
* 14:22 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic2032.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 15:15 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25746 and previous config saved to /var/cache/conftool/dbconfig/20220420-151509-kormat.json
* 14:22 root@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on logstash2035.codfw.wmnet with reason: pdu
* 15:14 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25745 and previous config saved to /var/cache/conftool/dbconfig/20220420-151419-kormat.json
* 14:22 godog: poweroff logstash2035 - [[phab:T310145|T310145]]
* 15:08 kormat@cumin1001: dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25744 and previous config saved to /var/cache/conftool/dbconfig/20220420-150806-kormat.json
* 14:22 root@cumin1001: START - Cookbook sre.hosts.downtime for 4:00:00 on logstash2035.codfw.wmnet with reason: pdu
* 15:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host serpens.wikimedia.org
* 14:21 Emperor: shutdown ms-be20[58,64].codfw.wmnet for PDU swap [[phab:T310145|T310145]]
* 15:05 kormat@cumin1001: dbctl commit (dc=all): 'es1034 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25743 and previous config saved to /var/cache/conftool/dbconfig/20220420-150539-kormat.json
* 14:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:05 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:04 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host serpens.wikimedia.org
* 14:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 15:02 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:14 Lucas_WMDE: UTC afternoon backport+config window done
* 15:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:13 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/CommonSettings-labs.php: Config: [[gerrit:820454{{!}}Remove unused $wgMathUseRestBase (T274436)]] (duration: 03m 01s)
* 15:02 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:02 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 15:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:05 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/CommonSettings-labs.php: Config: [[gerrit:820254{{!}}CommonSettings-labs: Fix usage of $wgSFSValidateIPListLocationMD5]] (duration: 02m 51s)
* 15:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25742 and previous config saved to /var/cache/conftool/dbconfig/20220420-150119-ladsgroup.json
* 14:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:00 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: updating wmf-puppet-dashboard for keystone authentication support [[phab:T274666|T274666]] (eqiad1) (duration: 05m 03s)
* 14:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:00 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25741 and previous config saved to /var/cache/conftool/dbconfig/20220420-150005-kormat.json
* 14:05 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic2033.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 14:59 kormat@cumin1001: dbctl commit (dc=all): 'db1178 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25740 and previous config saved to /var/cache/conftool/dbconfig/20220420-145915-kormat.json
* 14:04 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic2033.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 14:55 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6]: updating wmf-puppet-dashboard for keystone authentication support [[phab:T274666|T274666]] (eqiad1)
* 14:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:54 kormat@cumin1001: dbctl commit (dc=all): 'db1178 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25739 and previous config saved to /var/cache/conftool/dbconfig/20220420-145454-kormat.json
* 13:59 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/wikitech.php: Config: [[gerrit:820255{{!}}wikitech: Remove old LDAP config vars]] (duration: 02m 54s)
* 14:54 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1178.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:54 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1178.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:58 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be[2058,2064].codfw.wmnet with reason: PDU work
* 14:53 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codfw1dev) (duration: 02m 03s)
* 13:58 mvernon@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be[2058,2064].codfw.wmnet with reason: PDU work
* 14:52 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25738 and previous config saved to /var/cache/conftool/dbconfig/20220420-145223-kormat.json
* 13:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:51 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codfw1dev)
* 13:58 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:51 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad
* 13:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:50 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25737 and previous config saved to /var/cache/conftool/dbconfig/20220420-145057-kormat.json
* 13:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:47 kormat@cumin1001: dbctl commit (dc=all): 'es1025 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25735 and previous config saved to /var/cache/conftool/dbconfig/20220420-144730-kormat.json
* 14:47 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:47 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1025.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:46 kormat@cumin1001: dbctl commit (dc=all): 'es1022 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25734 and previous config saved to /var/cache/conftool/dbconfig/20220420-144615-kormat.json
* 14:46 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:46 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1022.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:46 kormat@cumin1001: dbctl commit (dc=all): 'es1034 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25733 and previous config saved to /var/cache/conftool/dbconfig/20220420-144557-kormat.json
* 14:46 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev) (duration: 01m 59s)
* 14:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1034.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:45 kormat@cumin1001: dbctl commit (dc=all): 'es1033 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25732 and previous config saved to /var/cache/conftool/dbconfig/20220420-144511-kormat.json
* 14:45 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:45 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1033.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:45 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25731 and previous config saved to /var/cache/conftool/dbconfig/20220420-144501-kormat.json
* 14:44 kormat@cumin1001: dbctl commit (dc=all): 'db1174 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25730 and previous config saved to /var/cache/conftool/dbconfig/20220420-144443-kormat.json
* 14:44 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:44 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1174.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:43 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev)
* 14:43 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25729 and previous config saved to /var/cache/conftool/dbconfig/20220420-144352-kormat.json
* 14:42 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25728 and previous config saved to /var/cache/conftool/dbconfig/20220420-144252-kormat.json
* 14:42 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25727 and previous config saved to /var/cache/conftool/dbconfig/20220420-144200-kormat.json
* 14:42 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25726 and previous config saved to /var/cache/conftool/dbconfig/20220420-144159-kormat.json
* 14:41 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25725 and previous config saved to /var/cache/conftool/dbconfig/20220420-144134-kormat.json
* 14:38 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
* 14:37 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
* 14:37 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25724 and previous config saved to /var/cache/conftool/dbconfig/20220420-143719-kormat.json
* 14:35 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25723 and previous config saved to /var/cache/conftool/dbconfig/20220420-143554-kormat.json
* 14:33 taavi@deploy1002: Finished deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev) (duration: 02m 08s)
* 14:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25722 and previous config saved to /var/cache/conftool/dbconfig/20220420-143258-ladsgroup.json
* 14:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 14:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 14:30 taavi@deploy1002: Started deploy [horizon/deploy@9d02cd6] (dev): updating wmf-puppet-dashboard for keystone authentication support (codwf1dev)
* 14:29 kormat@cumin1001: dbctl commit (dc=all): 'db1149 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25721 and previous config saved to /var/cache/conftool/dbconfig/20220420-142957-kormat.json
* 14:28 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25720 and previous config saved to /var/cache/conftool/dbconfig/20220420-142848-kormat.json
* 14:27 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25719 and previous config saved to /var/cache/conftool/dbconfig/20220420-142748-kormat.json
* 14:27 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25718 and previous config saved to /var/cache/conftool/dbconfig/20220420-142656-kormat.json
* 14:26 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25717 and previous config saved to /var/cache/conftool/dbconfig/20220420-142656-kormat.json
* 14:26 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25716 and previous config saved to /var/cache/conftool/dbconfig/20220420-142630-kormat.json
* 14:25 kormat@cumin1001: dbctl commit (dc=all): 'db1149 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25715 and previous config saved to /var/cache/conftool/dbconfig/20220420-142526-kormat.json
* 14:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1149.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1149.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:23 moritzm: installing webperf1004 [[phab:T305460|T305460]]
* 14:23 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25714 and previous config saved to /var/cache/conftool/dbconfig/20220420-142310-kormat.json
* 14:22 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25713 and previous config saved to /var/cache/conftool/dbconfig/20220420-142215-kormat.json
* 14:20 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25712 and previous config saved to /var/cache/conftool/dbconfig/20220420-142050-kormat.json
* 14:13 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25711 and previous config saved to /var/cache/conftool/dbconfig/20220420-141345-kormat.json
* 14:12 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25710 and previous config saved to /var/cache/conftool/dbconfig/20220420-141244-kormat.json
* 14:12 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2090.codfw.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:12 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2090.codfw.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:11 kormat@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25709 and previous config saved to /var/cache/conftool/dbconfig/20220420-141152-kormat.json
* 14:11 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25708 and previous config saved to /var/cache/conftool/dbconfig/20220420-141152-kormat.json
* 14:11 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25707 and previous config saved to /var/cache/conftool/dbconfig/20220420-141127-kormat.json
* 14:08 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25706 and previous config saved to /var/cache/conftool/dbconfig/20220420-140806-kormat.json
* 14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25705 and previous config saved to /var/cache/conftool/dbconfig/20220420-140711-kormat.json
* 14:05 kormat@cumin1001: dbctl commit (dc=all): 'db1180 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25704 and previous config saved to /var/cache/conftool/dbconfig/20220420-140546-kormat.json
* 14:01 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 14:01 kormat@cumin1001: dbctl commit (dc=all): 'db1180 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25703 and previous config saved to /var/cache/conftool/dbconfig/20220420-140123-kormat.json
* 14:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1180.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1180.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:01 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 14:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25702 and previous config saved to /var/cache/conftool/dbconfig/20220420-140105-ladsgroup.json
* 14:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 14:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 14:00 kormat@cumin1001: dbctl commit (dc=all): 'db1177 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25701 and previous config saved to /var/cache/conftool/dbconfig/20220420-140029-kormat.json
* 14:00 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1177.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:00 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1177.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:59 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25700 and previous config saved to /var/cache/conftool/dbconfig/20220420-135956-kormat.json
* 13:58 kormat@cumin1001: dbctl commit (dc=all): 'es1024 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25699 and previous config saved to /var/cache/conftool/dbconfig/20220420-135841-kormat.json
* 13:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25698 and previous config saved to /var/cache/conftool/dbconfig/20220420-135750-marostegui.json
* 13:57 kormat@cumin1001: dbctl commit (dc=all): 'es1021 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25697 and previous config saved to /var/cache/conftool/dbconfig/20220420-135740-kormat.json
* 13:56 kormat@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25696 and previous config saved to /var/cache/conftool/dbconfig/20220420-135648-kormat.json
* 13:56 kormat@cumin1001: dbctl commit (dc=all): 'db1127 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25695 and previous config saved to /var/cache/conftool/dbconfig/20220420-135623-kormat.json
* 13:54 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 100%: repooling [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25694 and previous config saved to /var/cache/conftool/dbconfig/20220420-135417-kormat.json
* 13:54 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25693 and previous config saved to /var/cache/conftool/dbconfig/20220420-135417-kormat.json
* 13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25692 and previous config saved to /var/cache/conftool/dbconfig/20220420-135302-kormat.json
* 13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:53 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:48 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784311{{!}}filebackend: Fix link to thumb url in testcommonswiki (T306139)]] (duration: 00m 53s)
* 13:48 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:820404{{!}}Remove unused $wgIncludejQueryMigrate (T280944)]] (2/2) (duration: 03m 03s)
* 13:44 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25691 and previous config saved to /var/cache/conftool/dbconfig/20220420-134452-kormat.json
* 13:45 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P25690 and previous config saved to /var/cache/conftool/dbconfig/20220420-134238-marostegui.json
* 13:45 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820404{{!}}Remove unused $wgIncludejQueryMigrate (T280944)]] (1/2) (duration: 02m 58s)
* 13:39 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 75%: repooling [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25689 and previous config saved to /var/cache/conftool/dbconfig/20220420-133914-kormat.json
* 13:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:39 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25688 and previous config saved to /var/cache/conftool/dbconfig/20220420-133913-kormat.json
* 13:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:38 kormat@cumin1001: dbctl commit (dc=all): 'db1148 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25687 and previous config saved to /var/cache/conftool/dbconfig/20220420-133757-kormat.json
* 13:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:36 kormat@cumin1001: dbctl commit (dc=all): 'es1024 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25686 and previous config saved to /var/cache/conftool/dbconfig/20220420-133622-kormat.json
* 13:40 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic2066.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 13:36 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:39 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic2066.codfw.wmnet with reason: [[phab:T310145|T310145]]
* 13:36 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1024.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:35 kormat@cumin1001: dbctl commit (dc=all): 'es1021 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25685 and previous config saved to /var/cache/conftool/dbconfig/20220420-133546-kormat.json
* 13:37 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:820402{{!}}Remove unused $wgLegacyJavaScriptGlobals (T72470)]] (2/2) (duration: 02m 59s)
* 13:35 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:37 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:35 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1021.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:33 kormat@cumin1001: dbctl commit (dc=all): 'db1148 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25684 and previous config saved to /var/cache/conftool/dbconfig/20220420-133317-kormat.json
* 13:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1148.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:34 lucaswerkmeister-wmde@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820402{{!}}Remove unused $wgLegacyJavaScriptGlobals (T72470)]] (1/2) (duration: 02m 58s)
* 13:33 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1148.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:30 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:30 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:30 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:30 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1127.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:30 kormat@cumin1001: dbctl commit (dc=all): 'db1147 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25683 and previous config saved to /var/cache/conftool/dbconfig/20220420-133000-kormat.json
* 13:29 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25682 and previous config saved to /var/cache/conftool/dbconfig/20220420-132948-kormat.json
* 13:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P25681 and previous config saved to /var/cache/conftool/dbconfig/20220420-132733-marostegui.json
* 13:24 kormat@cumin1001: dbctl commit (dc=all): 'es1032 (re)pooling @ 50%: repooling [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25680 and previous config saved to /var/cache/conftool/dbconfig/20220420-132410-kormat.json
* 13:24 kormat@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25679 and previous config saved to /var/cache/conftool/dbconfig/20220420-132409-kormat.json
* 13:23 kormat@cumin1001: dbctl commit (dc=all): 'es1031 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25678 and previous config saved to /var/cache/conftool/dbconfig/20220420-132325-kormat.json
* 13:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:23 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1031.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:14 kormat@cumin1001: dbctl commit (dc=all): 'db1147 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25677 and previous config saved to /var/cache/conftool/dbconfig/20220420-131456-kormat.json
* 13:14 kormat@cumin1001: dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25676 and previous config saved to /var/cache/conftool/dbconfig/20220420-131444-kormat.json
* 13:14 vgutierrez: restarting pybal on lvs1017
* 13:12 kormat@cumin1001: dbctl commit (dc=all): 'es1030 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25675 and previous config saved to /var/cache/conftool/dbconfig/20220420-131251-kormat.json
* 13:12 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:12 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1030.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25674 and previous config saved to /var/cache/conftool/dbconfig/20220420-131228-marostegui.json
* 13:12 kormat@cumin1001: dbctl commit (dc=all): 'Change es2 'master' to es1026 [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25673 and previous config saved to /var/cache/conftool/dbconfig/20220420-131222-kormat.json
* 13:11 vgutierrez: restarting pybal on lvs1018
* 13:10 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1168.eqiad.wmnet w