You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26857 and previous config saved to /var/cache/conftool/dbconfig/20220428-014723-ladsgroup.json)
imported>Stashbot
(eileen: civicrm upgraded from dcef393d to e198fb4c)
 
(148 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== 2022-04-28 ==
== 2022-09-27 ==
* 01:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26857 and previous config saved to /var/cache/conftool/dbconfig/20220428-014723-ladsgroup.json
* 01:17 eileen: civicrm upgraded from {{Gerrit|dcef393d}} to {{Gerrit|e198fb4c}}
* 01:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26856 and previous config saved to /var/cache/conftool/dbconfig/20220428-013218-ladsgroup.json
* 01:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2103 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34924 and previous config saved to /var/cache/conftool/dbconfig/20220927-011543-ladsgroup.json
* 01:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26855 and previous config saved to /var/cache/conftool/dbconfig/20220428-011007-ladsgroup.json
* 00:50 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1007.wikimedia.org
* 01:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 00:42 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudcontrol1006.wikimedia.org
* 01:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 00:40 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1007.wikimedia.org
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26854 and previous config saved to /var/cache/conftool/dbconfig/20220428-010935-ladsgroup.json
* 00:32 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudcontrol1005.wikimedia.org
* 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26853 and previous config saved to /var/cache/conftool/dbconfig/20220428-005430-ladsgroup.json
* 00:31 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.wikimedia.org
* 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26852 and previous config saved to /var/cache/conftool/dbconfig/20220428-003925-ladsgroup.json
* 00:16 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudcontrol1005.wikimedia.org
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26851 and previous config saved to /var/cache/conftool/dbconfig/20220428-002420-ladsgroup.json
* 00:15 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudnet1005.eqiad.wmnet
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26850 and previous config saved to /var/cache/conftool/dbconfig/20220428-002354-ladsgroup.json
* 00:15 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1005.eqiad.wmnet
* 00:15 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet
* 00:13 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudnet1005.eqiad.wmnet
* 00:10 razzi@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet
* 00:13 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1005.eqiad.wmnet
* 00:09 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34923 and previous config saved to /var/cache/conftool/dbconfig/20220927-000525-ladsgroup.json
* 00:09 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 00:09 razzi@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:04 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices1005.wikimedia.org
* 00:09 razzi@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on an-web1001.eqiad.wmnet with reason: Restart for kernel upgrade
* 00:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26849 and previous config saved to /var/cache/conftool/dbconfig/20220428-000849-ladsgroup.json
* 00:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 00:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26848 and previous config saved to /var/cache/conftool/dbconfig/20220428-000317-ladsgroup.json
* 00:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 00:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 00:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34922 and previous config saved to /var/cache/conftool/dbconfig/20220927-000434-ladsgroup.json
* 00:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1122.eqiad.wmnet with reason: Maintenance
* 00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26847 and previous config saved to /var/cache/conftool/dbconfig/20220428-000244-ladsgroup.json


== 2022-04-27 ==
== 2022-09-26 ==
* 23:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298563|T298563]])', diff saved to https://phabricator.wikimedia.org/P26846 and previous config saved to /var/cache/conftool/dbconfig/20220427-235953-ladsgroup.json
* 23:56 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudservices1005.wikimedia.org
* 23:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298558|T298558]])', diff saved to https://phabricator.wikimedia.org/P26845 and previous config saved to /var/cache/conftool/dbconfig/20220427-235700-ladsgroup.json
* 23:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P34921 and previous config saved to /var/cache/conftool/dbconfig/20220926-234928-ladsgroup.json
* 23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26844 and previous config saved to /var/cache/conftool/dbconfig/20220427-235344-ladsgroup.json
* 23:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P34920 and previous config saved to /var/cache/conftool/dbconfig/20220926-233422-ladsgroup.json
* 23:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26843 and previous config saved to /var/cache/conftool/dbconfig/20220427-234739-ladsgroup.json
* 23:34 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudservices1004.wikimedia.org
* 23:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P26842 and previous config saved to /var/cache/conftool/dbconfig/20220427-234448-ladsgroup.json
* 23:21 andrew@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudservices1004.wikimedia.org
* 23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26841 and previous config saved to /var/cache/conftool/dbconfig/20220427-234155-ladsgroup.json
* 23:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T314041|T314041]])', diff saved to https
* 23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26840 and previous config saved to /var/cache/conftool/dbconfig/20220427-233839-ladsgroup.json
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26839 and previous config saved to /var/cache/conftool/dbconfig/20220427-233628-ladsgroup.json
* 23:36 ladsgroup@cumin1001


== 2022-04-26 ==
== 2022-09-22 ==
* 23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P26648 and previous config saved to /var/cache/conftool/dbconfig/20220426-235422-ladsgroup.json
* 22:20 joal@deploy1002: Finished deploy [airflow-dags/analytics@901f810]: (no justification provided) (duration: 00m 11s)
* 23:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298554|T298554]])', diff saved to https://phabricator.wikimedia.org/P26647 and previous config saved to /var/cache/conftool/dbconfig/20220426-234224-ladsgroup.json
* 22:19 joal@deploy1002: Started deploy [airflow-dags/analytics@901f810]: (no justification provided)
* 23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 21:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 21:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 21:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 21:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 23:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26646 and previous config saved to /var/cache/conftool/dbconfig/20220426-234000-ladsgroup.json
* 21:23 dancy@deploy1002: backport aborted: (duration: 00m 05s)
* 23:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 20:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 23:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 20:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26645 and previous config saved to /var/cache/conftool/dbconfig/20220426-233953-ladsgroup.json
* 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26644 and previous config saved to /var/cache/conftool/dbconfig/20220426-233917-ladsgroup.json
* 20:55 brennen: end of utc late backport & config window
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26643 and previous config saved to /var/cache/conftool/dbconfig/20220426-233642-ladsgroup.json
* 20:55 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 20:54 brennen@deploy1002: Finished scap: Backport for [[gerrit:834364{{!}}Restrict figure to the size of the media (T305357 T318300)]], [[gerrit:834366{{!}}Fix media alignment since disabling wgParserEnableLegacyMediaDOM (T318300)]] (duration: 06m 33s)
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 20:53 joal@deploy1002: Finished deploy [airflow-dags/analytics@6c81e6f]: (no justification provided) (duration: 00m 10s)
* 23:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 14 hosts with reason: Maintenance
* 20:53 joal@deploy1002: Started deploy [airflow-dags/analytics@6c81e6f]: (no justification provided)
* 23:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 14 hosts with reason: Maintenance
* 20:48 brennen@deploy1002: brennen and arlolra: Backport for [[gerrit:834364{{!}}Restrict figure to the size of the media (T305357 T318300)]], [[gerrit:834366{{!}}Fix media alignment since disabling wgParserEnableLegacyMediaDOM (T318300)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 23:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 20:47 brennen@deploy1002: Started scap: Backport for [[gerrit:834364{{!}}Restrict figure to the size of the media (T305357 T318300)]], [[gerrit:834366{{!}}Fix media alignment since disabling wgParserEnableLegacyMediaDOM (T318300)]]
* 23:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance
* 20:36 brennen@deploy1002: backport aborted: (duration: 02m 16s)
* 23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26642 and previous config saved to /var/cache/conftool/dbconfig/20220426-233545-ladsgroup.json
* 23:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26641 and previous config saved to /var/cache/conftool/dbconfig/20220426-232447-ladsgroup.json
* 23:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26640 and previous config saved to /var/cache/conftool/dbconfig/20220426-232040-ladsgroup.json
* 23:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P26639 and previous config saved to /var/cache/conftool/dbconfig/20220426-230942-ladsgroup.json
* 23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P26638 and previous config saved to /var/cache/conftool/dbconfig/20220426-230535-ladsgroup.json
* 22:57 tgr@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/extension.json: Backport: [[gerrit:786366{{!}}Enable SkinAddFooterLinks hook]] (duration: 00m 51s)
* 22:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26637 and previous config saved to /var/cache/conftool/dbconfig/20220426-225437-ladsgroup.json
* 22:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3316 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26636 and previous config saved to /var/cache/conftool/dbconfig/20220426-225326-ladsgroup.json
* 22:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26635 and previous config saved to /var/cache/conftool/dbconfig/20220426-225030-ladsgroup.json
* 22:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26634 and previous config saved to /var/cache/conftool/dbconfig/20220426-224757-ladsgroup.json
* 22:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26633 and previous config saved to /var/cache/conftool/dbconfig/20220426-224749-ladsgroup.json
* 22:46 tgr@deploy1002: Finished scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]] (duration: 21m 40s)
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26632 and previous config saved to /var/cache/conftool/dbconfig/20220426-223244-ladsgroup.json
* 22:25 tgr@deploy1002: Started scap: backport with i18n changes: [[gerrit:785944]], [[gerrit:785941]]
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P26631 and previous config saved to /var/cache/conftool/dbconfig/20220426-221739-ladsgroup.json
* 22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26630 and previous config saved to /var/cache/conftool/dbconfig/20220426-220234-ladsgroup.json
* 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26629 and previous config saved to /var/cache/conftool/dbconfig/20220426-220001-ladsgroup.json
* 22:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 22:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
* 21:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26628 and previous config saved to /var/cache/conftool/dbconfig/20220426-215953-ladsgroup.json
* 21:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26627 and previous config saved to /var/cache/conftool/dbconfig/20220426-214448-ladsgroup.json
* 21:38 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9] (duration: 00m 07s)
* 21:37 aqu@deploy1002: Started deploy [airflow-dags/analytics@e5fecc9]: Fix typo in mediarequest/hourly sensor [airflow-dags/analytics@e5fecc9]
* 21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P26626 and previous config saved to /var/cache/conftool/dbconfig/20220426-212943-ladsgroup.json
* 21:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26625 and previous config saved to /var/cache/conftool/dbconfig/20220426-211437-ladsgroup.json
* 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26624 and previous config saved to /var/cache/conftool/dbconfig/20220426-211204-ladsgroup.json
* 21:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 21:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
* 21:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26623 and previous config saved to /var/cache/conftool/dbconfig/20220426-211156-ladsgroup.json
* 21:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2418.codfw.wmnet
* 21:03 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2417.codfw.wmnet
* 21:02 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2416.codfw.wmnet
* 21:01 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|cab00628da0ba6226ff162cfc848bea35a35783a}}: fix wmgVectorMaxWidthOptionsNamespaces ([[phab:T300182|T300182]]) (duration: 01m 00s)
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:00 mutante: mw2416, mw2417, mw2418 - scap pull
* 20:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26621 and previous config saved to /var/cache/conftool/dbconfig/20220426-205651-ladsgroup.json
* 20:50 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 07s)
* 20:50 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
* 20:49 urbanecm@deploy1002: Synchronized wmf-config/SearchSettingsForWikidata.php: {{Gerrit|f76bc806157a3f4c88d44cd467de347b4b471f4e}}: Correct wbsearchentities profiles (duration: 00m 57s)
* 20:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P26620 and previous config saved to /var/cache/conftool/dbconfig/20220426-204146-ladsgroup.json
* 20:40 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2415.codfw.wmnet
* 20:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2414.codfw.wmnet
* 20:39 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2413.codfw.wmnet
* 20:38 dzahn@cumin2002: conftool action : set/weight=30; selector: dc=codfw,name=mw2412.codfw.wmnet
* 20:37 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.9/skins/Vector/resources/skins.vector.styles/: {{Gerrit|019a812176bb940383ddeb22f8a74b5d0f447bf1}}: [ToC] Increase threshold for ToC collapsing to 1000px ([[phab:T306904|T306904]]) (duration: 00m 50s)
* 20:36 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/skins/Vector/resources/skins.vector.styles/: {{Gerrit|31ed884d6eda998f8625a88be0f4aa5fd67aef4b}}: [ToC] Increase threshold for ToC collapsing to 1000px ([[phab:T306904|T306904]]) (duration: 00m 50s)
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:33 mutante: mw2412, mw2413, mw2414, mw2415 - scap pull, get into production the first time
* 20:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:25 brennen@deploy1002: Finished scap: Backport for [[gerrit:833817{{!}}Drops JS-side creation of "Source" link (T318266)]] (duration: 06m 09s)
* 20:29 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|fe0e119ef7c768373db4afed21537f85004a8ae2}}: Expand max-width to login, create account, disable on Wikidata ([[phab:T300182|T300182]], [[phab:T306834|T306834]]; 2/2) (duration: 00m 54s)
* 20:19 brennen@deploy1002: brennen and tpt: Backport for [[gerrit:833817{{!}}Drops JS-side creation of "Source" link (T318266)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 20:28 aqu@deploy1002: Finished deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87] (duration: 00m 17s)
* 20:19 brennen@deploy1002: Started scap: Backport for [[gerrit:833817{{!}}Drops JS-side creation of "Source" link (T318266)]]
* 20:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|fe0e119ef7c768373db4afed21537f85004a8ae2}}: Expand max-width to login, create account, disable on Wikidata ([[phab:T300182|T300182]], [[phab:T306834|T306834]]; 1/2) (duration: 00m 56s)
* 20:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:27 aqu@deploy1002: Started deploy [airflow-dags/analytics@e177d87]: Bump jar dependency to 0.1.27 in mediarequest/hourly [airflow-dags/analytics@e177d87]
* 20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26619 and previous config saved to /var/cache/conftool/dbconfig/20220426-202641-ladsgroup.json
* 20:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|e3ce97b97c1d83dc4f538040da92a571895cb4d0}}: Enable table of contents a/b test on euwiki and hewiki, enable reading depth ([[phab:T306606|T306606]]) (duration: 00m 52s)
* 20:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:24 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26618 and previous config saved to /var/cache/conftool/dbconfig/20220426-202407-ladsgroup.json
* 20:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 20:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 20:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26617 and previous config saved to /var/cache/conftool/dbconfig/20220426-202359-ladsgroup.json
* 20:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298556|T298556]])', diff saved to https://phabricator.wikimedia.org/P26616 and previous config saved to /var/cache/conftool/dbconfig/20220426-201610-ladsgroup.json
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:11 urbanecm@deploy1002: Synchronized wmf-config/: {{Gerrit|9805e61f7006edf45199a3e22494945bffaaeb4d}}: Add wbsearchentities profiles for testing ([[phab:T306644|T306644]]) (duration: 00m 53s)
* 19:45 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
* 20:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:38 jhuneidi@deploy1002: Started scap: testing
* 20:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164', diff saved to https://phabricator.wikimedia.org/P26615 and previous config saved to /var/cache/conftool/dbconfig/20220426-200854-ladsgroup.json
* 18:38 dancy@deploy1002: Started scap: testing
* 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:37 jhuneidi@deploy1002: Started scap: testing
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:34 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@265686e]: (no justification provided) (duration: 00m 13s)
* 20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:33 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@265686e]: (no justification provided)
* 20:05 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|080b8fc573d9d682038e09a7a7ad875bce478c00}}: cirrus: Turn
* 18:29 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.2  refs [[phab:T314191|T314191]]
* 18:23 dancy@deploy1002: Unlocked for deployment [ALL REPOSITORIES]: testing (duration: 00m 02s)
* 18:23 dancy@deploy1002: Locking from deployment [ALL REPOSITORIES]: testing (planned duration: 60m 00s)
*
* 13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:36 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1011 as pc1 primary [[phab:T306892|T306892]] (duration: 01m 37s)
* 13:25 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@0e9fb6b]: (no justification provided) (duration: 00m 11s)
* 13:29 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 13:25 nokafor@deploy1002: Started deploy [airflow-dags/analytics@0e9fb6b]: (no justification provided)
* 13:29 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 13:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:28 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 13:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:23 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on pc1011.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:23 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on pc1011.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:22 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Rebooting pc1011 [[phab:T306892|T306892]]
* 13:21 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on pc[2011,2014].codfw.wmnet,pc[1011,1014].eqiad.wmnet with reason: Rebooting pc1011 [[phab:T306892|T306892]]
* 13:21 kormat@deploy1002: Synchronized wmf-config/ProductionServices.php: Set pc1014 as pc1 primary [[phab:T306892|T306892]] (duration: 01m 07s)
* 13:18 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:18 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:14 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1005.eqiad.wmnet
* 13:08 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0b55db6f80df5f4c89f969332a6b31077a7172c4}}: Enable Tech Wishes survey on dewiki ([[phab:T316676|T316676]]) (duration: 04m 12s)
* 13:07 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1005.eqiad.wmnet
* 09:58 jbond@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts sretest1002.eqiad.wmnet
* 12:55 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid1004.eqiad.wmnet
* 09:27 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1002.eqiad.wmnet
* 12:52 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26550 and previous config saved to /var/cache/conftool/dbconfig/20220426-125244-kormat.json
* 08:46 awight@deploy1002: Finished deploy [kartotherian/deploy@4759a78]: Merge "Update kartotherian to e3f3854" (duration: 02m 27s)
* 12:48 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host druid1004.eqiad.wmnet
* 08:43 awight@deploy1002: Started deploy [kartotherian/deploy@4759a78]: Merge "Update kartotherian to e3f3854"
* 12:46 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org
* 08:35 hashar: Restarted CI Jenkins for plugin update
* 12:44 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org
* 08:33 jbond@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts sretest1002.eqiad.wmnet
* 12:37 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26547 and previous config saved to /var/cache/conftool/dbconfig/20220426-123740-kormat.json
* 08:33 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1002.eqiad.wmnet
* 12:24 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1015.eqiad.wmnet
* 07:18 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:832993{{!}}testwiki: Enable Section Translation on haw, la, ps and, xh Wikipedias (T317289)]] (duration: 03m 46s)
* 12:22 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26546 and previous config saved to /var/cache/conftool/dbconfig/20220426-122235-kormat.json
* 07:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 12:14 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1015.eqiad.wmnet
* 07:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 12:07 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26545 and previous config saved to /var/cache/conftool/dbconfig/20220426-120731-kormat.json
* 07:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 12:07 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26544 and previous config saved to /var/cache/conftool/dbconfig/20220426-120727-kormat.json
* 07:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 12:03 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 07:10 kart_: Updated cxserver to 2022-09-15-113346-production ([[phab:T317289|T317289]], [[phab:T315209|T315209]])
* 12:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti-test2001.codfw.wmnet to ganeti-test01.svc.codfw.wmnet
* 07:08 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 12:00 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup1001.eqiad.wmnet with OS bullseye
* 07:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 11:52 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26543 and previous config saved to /var/cache/conftool/dbconfig/20220426-115223-kormat.json
* 07:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 11:49 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup1001.eqiad.wmnet with reason: host reimage
* 07:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 11:46 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup1001.eqiad.wmnet with reason: host reimage
* 07:07 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 11:42 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup2001.codfw.wmnet with OS bullseye
* 07:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 11:37 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26542 and previous config saved to /var/cache/conftool/dbconfig/20220426-113719-kormat.json
* 07:06 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 11:34 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host ms-backup1001.eqiad.wmnet with OS bullseye
* 07:05 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 11:30 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup2001.codfw.wmnet with reason: host reimage
* 07:03 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 11:27 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup2001.codfw.wmnet with reason: host reimage
* 07:02 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 11:23 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti-test2001.codfw.wmnet
* 04:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 11:22 topranks: Reconfigre routing policy lsw1-f1-eqiad, rename policies to use lower-case
* 04:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 11:22 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26541 and previous config saved to /var/cache/conftool/dbconfig/20220426-112215-kormat.json
* 04:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 11:17 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3317 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26540 and previous config saved to /var/cache/conftool/dbconfig/20220426-111751-kormat.json
* 03:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 11:17 kormat@cumin1001: dbctl commit (dc=all): 'db1170:3312 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26539 and previous config saved to /var/cache/conftool/dbconfig/20220426-111741-kormat.json
* 03:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 11:17 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1170.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 03:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 11:17 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1170.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 03:45 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 11:16 topranks: Reconfigre routing policy lsw1-e1-eqiad, rename policies to use lower-case
* 03:40 mwpresync@deploy1002: Pruned MediaWiki: 1.39.0-wmf.28 (duration: 02m 02s)
* 11:13 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host ms-backup2001.codfw.wmnet with OS bullseye
* 03:38 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.2  refs [[phab:T314191|T314191]] (duration: 36m 08s)
* 11:11 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup1002.eqiad.wmnet with OS bullseye
* 03:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 11:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 03:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 11:09 topranks: Reconfigre routing policy lsw1-e2-eqiad, rename policies to use lower-case
* 03:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 11:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1014.eqiad.wmnet
* 03:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 11:08 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26538 and previous config saved to /var/cache/conftool/dbconfig/20220426-110819-kormat.json
* 03:05 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 11:05 topranks: Reconfigre routing policy lsw1-f2-eqiad, rename policies to use lower-case
* 03:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.2  refs [[phab:T314191|T314191]]
* 11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1014.eqiad.wmnet
* 02:42 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
* 11:00 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup1002.eqiad.wmnet with reason: host reimage
* 02:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 10:57 topranks: Reconfigre routing policy lsw1-e3-eqiad, rename policies to use lower-case
* 02:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 10:57 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup1002.eqiad.wmnet with reason: host reimage
* 02:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 10:53 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 02:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 10:53 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26537 and previous config saved to /var/cache/conftool/dbconfig/20220426-105315-kormat.json
* 02:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 10:45 jynus@cumin1001: START - Cookbook sre.hosts.reimage for host ms-backup1002.eqiad.wmnet with OS bullseye
* 02:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 10:44 topranks: Reconfigre routing policy lsw1-f3-eqiad, rename policies to use lower-case
* 02:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 10:43 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-backup2002.codfw.wmnet with OS bullseye
* 02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 10:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti-test2001.codfw.wmnet with OS bullseye
* 10:38 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26536 and previous config saved to /var/cache/conftool/dbconfig/20220426-103811-kormat.json
* 10:33 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 10:32 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-backup2002.codfw.wmnet with reason: host reimage
* 10:28 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-backup2002.codfw.wmnet with reason: host reimage
* 10:25 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1013.eqiad.wmnet
* 10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26535 and previous config saved to /var/cache/conftool/dbconfig/20220426-102307-kormat.json
* 10:23 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26534 and previous config saved to /var/cache/conftool/dbconfig/20220426-102303-kormat.json
* 10:15 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1013.eqiad.wmnet
* 10:14 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host ms-backup2002.codfw.wmnet with OS bullseye
* 10:08 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26533 and previous config saved to /var/cache/conftool/dbconfig/20220426-100758-kormat.json
* 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti-test2001.codfw.wmnet with reason: host reimage
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1122 into API', diff saved to https://phabricator.wikimedia.org/P26532 and previous config saved to /var/cache/conftool/dbconfig/20220426-100031-marostegui.json
* 10:00 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2001.codfw.wmnet with reason: host reimage
* 09:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P26531 and previous config saved to /var/cache/conftool/dbconfig/20220426-095957-root.json
* 09:56 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26530 and previous config saved to /var/cache/conftool/dbconfig/20220426-095627-kormat.json
* 09:52 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26528 and previous config saved to /var/cache/conftool/dbconfig/20220426-095254-kormat.json
* 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1002.eqiad.wmnet
* 09:51 nokafor@deploy1002: Finished deploy [airflow-dags/analytics@9dbd5bc]: (no justification provided) (duration: 00m 07s)
* 09:51 nokafor@deploy1002: Started deploy [airflow-dags/analytics@9dbd5bc]: (no justification provided)
* 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1002.eqiad.wmnet
* 09:47 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1012.eqiad.wmnet
* 09:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti-test2001.codfw.wmnet with OS bullseye
* 09:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P26526 and previous config saved to /var/cache/conftool/dbconfig/20220426-094453-root.json
* 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf1001.eqiad.wmnet
* 09:41 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26525 and previous config saved to /var/cache/conftool/dbconfig/20220426-094123-kormat.json
* 09:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf1001.eqiad.wmnet
* 09:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 09:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 09:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 09:37 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26524 and previous config saved to /var/cache/conftool/dbconfig/20220426-093750-kormat.json
* 09:36 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1012.eqiad.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf2002.codfw.wmnet
* 09:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 09:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 09:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 09:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 09:33 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3314 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26523 and previous config saved to /var/cache/conftool/dbconfig/20220426-093314-kormat.json
* 09:33 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:32 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:32 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 09:32 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 09:31 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf2002.codfw.wmnet
* 09:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host webperf2001.codfw.wmnet
* 09:29 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 09:29 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P26522 and previous config saved to /var/cache/conftool/dbconfig/20220426-092949-root.json
* 09:29 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 09:27 kevinbazira@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host webperf2001.codfw.wmnet
* 09:26 kevinbazira@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 09:26 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26521 and previous config saved to /var/cache/conftool/dbconfig/20220426-092619-kormat.json
* 09:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1115.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1115.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:25 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2093.codfw.wmnet,dborch1001.wikimedia.org with reason: Rebooting db1115 [[phab:T303174|T303174]]
* 09:25 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on db2093.codfw.wmnet,dborch1001.wikimedia.org with reason: Rebooting db1115 [[phab:T303174|T303174]]
* 09:23 topranks: Reconfigure CR routers following bgp policy changes (no-op) CR785284
* 09:23 mvernon@cumin1001: conftool action : set/pooled=yes; selector: service=swift-fe,name=ms-fe1012.eqiad.wmnet
* 09:23 mvernon@cumin1001: conftool action : set/pooled=yes; selector: service=nginx,name=ms-fe1012.eqiad.wmnet
* 09:20 mvernon@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1012.eqiad.wmnet
* 09:14 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P26520 and previous config saved to /var/cache/conftool/dbconfig/20220426-091445-root.json
* 09:14 mvernon@cumin1001: START - Cookbook sre.hosts.reboot-single for host ms-fe1012.eqiad.wmnet
* 09:11 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26519 and previous config saved to /var/cache/conftool/dbconfig/20220426-091115-kormat.json
* 09:11 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26518 and previous config saved to /var/cache/conftool/dbconfig/20220426-091111-kormat.json
* 09:10 kormat@cumin1001: dbctl commit (dc=all): 'db1146:3312 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26517 and previous config saved to /var/cache/conftool/dbconfig/20220426-091015-kormat.json
* 09:10 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:10 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1146.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:59 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P26516 and previous config saved to /var/cache/conftool/dbconfig/20220426-085941-root.json
* 08:56 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26515 and previous config saved to /var/cache/conftool/dbconfig/20220426-085607-kormat.json
* 08:47 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 08:44 marostegui@cumin1001: dbctl commit (dc=all): 'db1122 (re)pooling @ 1%: After reimage', diff saved to https://phabricator.wikimedia.org/P26514 and previous config saved to /var/cache/conftool/dbconfig/20220426-084437-root.json
* 08:43 jelto: pool name=mw229[7-9].codfw.wmnet, manual icinga recheck green after reboot
* 08:43 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw229[7-9].codfw.wmnet
* 08:41 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26513 and previous config saved to /var/cache/conftool/dbconfig/20220426-084103-kormat.json
* 08:34 moritzm: installing testvm2004 [[phab:T306499|T306499]]
* 08:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1122.eqiad.wmnet with OS bullseye
* 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1002.eqiad.wmnet
* 08:31 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 08:26 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26512 and previous config saved to /var/cache/conftool/dbconfig/20220426-082559-kormat.json
* 08:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host deploy1002.eqiad.wmnet
* 08:22 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3316 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26511 and previous config saved to /var/cache/conftool/dbconfig/20220426-082210-kormat.json
* 08:21 kormat@cumin1001: dbctl commit (dc=all): 'db1113:3315 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26510 and previous config saved to /var/cache/conftool/dbconfig/20220426-082155-kormat.json
* 08:21 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1113.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:21 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1113.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:19 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1122.eqiad.wmnet with reason: host reimage
* 08:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1122.eqiad.wmnet with reason: host reimage
* 08:08 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1122.eqiad.wmnet with OS bullseye
* 08:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2004.codfw.wmnet
* 08:03 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2289.codfw.wmnet
* 07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2288.codfw.wmnet
* 07:56 jelto@cumin1001: conftool action : set/pooled=yes; selector: name=mw2287.codfw.wmnet
* 07:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:48 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 07:47 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 07:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1003.wikimedia.org
* 07:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3005.wikimedia.org
* 07:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 ([[phab:T302185|T302185]])', diff saved to https://phabricator.wikimedia.org/P26509 and previous config saved to /var/cache/conftool/dbconfig/20220426-073627-ladsgroup.json
* 07:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast3005.wikimedia.org
* 07:29 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host bast1003.wikimedia.org
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P26508 and previous config saved to /var/cache/conftool/dbconfig/20220426-072122-ladsgroup.json
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109', diff saved to https://phabricator.wikimedia.org/P26507 and previous config saved to /var/cache/conftool/dbconfig/20220426-070617-ladsgroup.json
* 07:01 marostegui: dbmaint s2@eqiad [[phab:T298554|T298554]]
* 06:54 jayme@deploy1002: Finished deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided) (duration: 03m 05s)
* 06:51 jayme@deploy1002: Started deploy [restbase/deploy@0205f1d] (dev-cluster): (no justification provided)
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 ([[phab:T302185|T302185]])', diff saved to https://phabricator.wikimedia.org/P26506 and previous config saved to /var/cache/conftool/dbconfig/20220426-065112-ladsgroup.json
* 06:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1109.eqiad.wmnet with OS bullseye
* 06:45 jayme: imported scap 4.7.0 to stretch-/buster-/bullseye-wikimedia - [[phab:T306827|T306827]]
* 06:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1109.eqiad.wmnet with reason: host reimage
* 06:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1109.eqiad.wmnet with reason: host reimage
* 06:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1109.eqiad.wmnet with OS bullseye
* 06:16 marostegui: dbmaint s2@eqiad [[phab:T300381|T300381]]
* 06:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1109 ([[phab:T302185|T302185]])', diff saved to https://phabricator.wikimedia.org/P26505 and previous config saved to /var/cache/conftool/dbconfig/20220426-061519-ladsgroup.json
* 06:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1109.eqiad.wmnet with reason: Maintenance
* 06:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1109.eqiad.wmnet with reason: Maintenance
* 06:14 marostegui: dbmaint s2@eqiad [[phab:T298557|T298557]]
* 06:07 marostegui@cumin1001: dbctl commit (dc=all): 'Remove db1100, s5 master from API', diff saved to https://phabricator.wikimedia.org/P26504 and previous config saved to /var/cache/conftool/dbconfig/20220426-060734-marostegui.json
* 06:06 marostegui@cumin1001: dbctl commit (dc=all): 'db1162 is current s2 master, should not be in API [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26503 and previous config saved to /var/cache/conftool/dbconfig/20220426-060602-marostegui.json
* 06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1122 [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26502 and previous config saved to /var/cache/conftool/dbconfig/20220426-060344-root.json
* 06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1162 to s2 primary and set section read-write [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26501 and previous config saved to /var/cache/conftool/dbconfig/20220426-060058-marostegui.json
* 06:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26500 and previous config saved to /var/cache/conftool/dbconfig/20220426-060033-marostegui.json
* 06:00 marostegui: Starting s2 eqiad failover from db1122 to db1162 - [[phab:T306417|T306417]]
* 04:54 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1162 with weight 0 [[phab:T306417|T306417]]', diff saved to https://phabricator.wikimedia.org/P26498 and previous config saved to /var/cache/conftool/dbconfig/20220426-045406-root.json
* 04:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s2 [[phab:T306417|T306417]]
* 04:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s2 [[phab:T306417|T306417]]
* 04:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 04:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 04:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 04:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply


== 2022-04-25 ==
== 2022-09-19 ==
* 23:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:59 ebernhardson: [[phab:T317200|T317200]] start cirrussearch in-place reindex process for eqiad, codfw and cloudelastic
* 23:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:21 maryum: Deployed security patch for [[phab:T302479|T302479]]
* 23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:21 mstyles@deploy1002: Synchronized php-1.40.0-wmf.1/extensions/Translate/src/: (no justification provided) (duration: 03m 40s)
* 23:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:15 sbassett: Deployed security patch for [[phab:T312820|T312820]]
* 23:01 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785948{{!}}ActorMigration: Read from rev_actor field in all of small wikis (T275246)]] (duration: 00m 57s)
* 21:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:59 cjming: end of UTC late backport window
* 22:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:59 ebernhardson@deploy1002: Synchronized php-1.40.0-wmf.1/extensions/CirrusSearch/includes/Maintenance/MappingConfigBuilder.php: Backport: [[gerrit:833031{{!}}Add token_count subfield to outgoing_link (T317546)]] (duration: 03m 51s)
* 22:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:55 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:49 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:612348{{!}}TimedMediaHandler: Make videojs the only player on all group1 (T248418)]] (duration: 00m 54s)
* 20:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:04 dancy@deploy1002: Synchronized README: testing scap mods (duration: 00m 54s)
* 20:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:00 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudweb2001-dev.wikimedia.org
* 20:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:56 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:52 eileen: civicrm revision {{Gerrit|7de7ddd4}} -> {{Gerrit|a841cf55}}
* 20:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:49 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 21:43 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudweb2001-dev.wikimedia.org
* 21:40 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
* 21:38 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:35 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 21:26 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudcephmon[2002-2003]-dev.codfw.wmnet
* 21:25 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785919{{!}}[Web scroll] Restore original sampling rate (T305442)]] (duration: 01m 01s)
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:59 dzahn@cumin2002: conftool action : set/pooled=inactive; selector: dc=codfw,name=mw2286.codfw.wmnet
* 20:58 mutante: rebooting mw2415
* 20:27 catrope@deploy1002: Synchronized wmf-config/wikitech.php: Config: [[gerrit:785399{{!}}labtestwiki: update labtest ldap server (T304881)]] (duration: 01m 39s)
* 20:23 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785908{{!}}Enable TOC for all users opted into modern Vector outside of pilot wikis (T306608)]] (duration: 01m 40s)
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:21 jforrester@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:820459{{!}}Wikifunctions: Drop two config items moved to docker]] (duration: 03m 38s)
* 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:21 bking@cumin2002: START - Cookbook sre.wdqs.data-reload
* 20:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:57 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 7 hosts with reason: fresh role user
* 20:16 jforrester@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:829877{{!}}ExtensionDistributor: Add REL1_39 (T313925)]] (duration: 03m 38s)
* 19:57 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on 7 hosts with reason: fresh role user
* 20:12 cjming@deploy1002: Finished scap: Backport for [[gerrit:832715{{!}}Disable wgParserEnableLegacyMediaDOM on cswiki (T314318)]] (duration: 06m 31s)
* 19:53 mutante: rebooting mw2414 through mw2419
* 20:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:46 dancy@deploy1002: Finished scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 12m 54s)
* 20:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:43 mutante: rebooting mw2412, mw2413
* 20:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:34 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
* 20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:34 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on mw2412.codfw.wmnet with reason: fresh role user
* 20:06 cjming@deploy1002: cjming and arlolra: Backport for [[gerrit:832715{{!}}Disable wgParserEnableLegacyMediaDOM on cswiki (T314318)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 19:33 dancy@deploy1002: Started scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]]
* 20:06 cjming@deploy1002: Started scap: Backport for [[gerrit:832715{{!}}Disable wgParserEnableLegacyMediaDOM on cswiki (T314318)]]
* 19:30 dancy@deploy1002: Started scap: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]]
* 19:33 bking@cumin2002: END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97)
* 19:29 dancy@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 01m 43s)
* 19:33 bking@cumin2002: START - Cookbook sre.wdqs.data-reload
* 19:27 dancy@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Config: [[gerrit:781060{{!}}Improve support for realms other than production and labs]] (duration: 01m 42s)
* 19:33 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
* 19:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:30 bking@cumin2002: START - Cookbook sre.wdqs.data-reload
* 19:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:30 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
* 19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:30 bking@cumin2002: START - Cookbook sre.wdqs.data-reload
* 19:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:43 dancy@deploy1002: Installation of scap version "4.21.0" completed for 561 hosts
* 19:09 mutante: turning mw2412 through mw2419 into actual appservers - applying roles for the first time, will cause alerts probably
* 17:42 dancy@deploy1002: Installing scap version "4.21.0" for 561 hosts
* 19:09 cwhite: install grafana-plugins 0.5 and restart grafana on grafana1002 [[phab:T304583|T304583]]
* 17:36 dancy@deploy1002: Sync cancelled.
* 18:47 cstone: payments-wiki revision changed from {{Gerrit|a3c69385}} to {{Gerrit|786dc94f}}
* 17:36 dancy@deploy1002: dancy: testing, disregard synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 17:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26497 and previous config saved to /var/cache/conftool/dbconfig/20220425-175957-ladsgroup.json
* 17:36 dancy@deploy1002: Started scap: testing, disregard
* 17:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26496 and previous config saved to /var/cache/conftool/dbconfig/20220425-174451-ladsgroup.json
* 14:03 urbanecm: Purge https://en.wikipedia.org/static/images/project-logos/ukwikivoyage<nowiki>{</nowiki>.png,-1.5x.png,-2x.png<nowiki>}</nowiki> ([[phab:T317718|T317718]])
* 17:39 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudservices[2002-2003]-dev.wikimedia.org
* 14:02 urbanecm@deploy1002: Synchronized static/images/project-logos/: {{Gerrit|6c7151d969b6997bd9cce042b7bc78c282dd9b26}}: Regenerate ukwikivoyage logo ([[phab:T317718|T317718]]) (duration: 03m 46s)
* 17:34 andrew@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:30 andrew@cumin1001: START - Cookbook sre.dns.netbox
* 13:59 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26495 and previous config saved to /var/cache/conftool/dbconfig/20220425-172946-ladsgroup.json
* 13:59 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:24 andrew@cumin1001: START - Cookbook sre.hosts.decommission for hosts cloudservices[2002-2003]-dev.wikimedia.org
* 13:58 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:17 aokoth@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM otrs1001.eqiad.wmnet
* 13:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26494 and previous config saved to /var/cache/conftool/dbconfig/20220425-171441-ladsgroup.json
* 13:17 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|cbf161d148228e0e706813f923ab1a5d4b42757a}}: GrowthExperiments: Enable image recommendations for el/pl/zh/id/ro ([[phab:T314518|T314518]]) (duration: 04m 01s)
* 17:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26493 and previous config saved to /var/cache/conftool/dbconfig/20220425-171223-ladsgroup.json
* 17:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 17:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 17:04 aokoth@cumin1001: START - Cookbook sre.ganeti.reboot-vm for VM otrs1001.eqiad.wmnet
* 16:47 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-eqiad cluster: Reboot kafka nodes
* 15:56 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
* 15:56 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
* 15:54 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1001.eqiad.wmnet
* 15:47 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki1001.eqiad.wmnet
* 15:46 jbond@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki.discovery.wmnet on all recursors
* 15:46 jbond@cumin1001: START - Cookbook sre.dns.wipe-cache pki.discovery.wmnet on all recursors
* 15:41 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:37 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-eqiad cluster: Reboot kafka nodes
* 15:36 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 15:32 herron@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka logging-codfw cluster: Reboot kafka nodes
* 15:25 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 15:23 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:21 jbond@cumin1001: START - Cookbook sre.dns.netbox
* 15:20 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki-root1001.eqiad.wmnet
* 15:18 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki2001.codfw.wmnet
* 15:16 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki-root1001.eqiad.wmnet
* 15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 15:15 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
* 15:15 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 15:15 jelto@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 15:13 jbond@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host pki2001.codfw.wmnet
* 15:12 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host pki2001.codfw.wmnet
* 15:10 jbond@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "test sync - jbond@cumin1001"
* 15:09 jbond@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test sync - jbond@cumin1001"
* 14:52 krinkle@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|I22240af06d}} (duration: 01m 42s)
* 14:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:43 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:21 herron@cumin1001: START - Cookbook sre.kafka.reboot-workers for Kafka logging-codfw cluster: Reboot kafka nodes
* 14:13 jelto: mw2253: remove puppet lock of stuck puppet run due to reboot, run-puppet-agent
* 14:10 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host aqs1011.eqiad.wmnet
* 14:00 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1011.eqiad.wmnet
* 13:51 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1010.eqiad.wmnet
* 13:41 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs1010.eqiad.wmnet
* 13:39 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1003.eqiad.wmnet
* 13:35 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1003.eqiad.wmnet
* 13:31 jelto: maintenance (rolling reboot) on api_appserver in codfw (cookbook sre.hosts.reboot-cluster -D codfw -c api_appserver --percentage 5 --grace_sleep 60)
* 13:30 jelto@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 13:28 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1002.eqiad.wmnet
* 13:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1002.eqiad.wmnet
* 13:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26492 and previous config saved to /var/cache/conftool/dbconfig/20220425-131411-ladsgroup.json
* 13:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:13 urbanecm: UTC afternoon B&C window done
* 07:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:12 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/CentralAuth/includes/User/GlobalUserSelectQueryBuilder.php: {{Gerrit|c4c4c3219ad69705c83caa50754d95285c96f352}}: GlobalUserSelectQueryBuilder: Do not fatal when no users are returned ([[phab:T306535|T306535]]) (duration: 00m 54s)
* 13:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:04 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0338c9bb9ff6ad388270d1e2df5013d8a49d1210}}: GrowthExperiments: Do not use facebook in campaign pattern ([[phab:T303785|T303785]]) (duration: 00m 51s)
* 13:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2001.wikimedia.org
* 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26491 and previous config saved to /var/cache/conftool/dbconfig/20220425-125906-ladsgroup.json
* 12:58 krinkle@deploy1002: Synchronized private/PrivateSettings.php: {{Gerrit|If4d7ea24b4db59}} (duration: 00m 59s)
* 12:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2001.wikimedia.org
* 12:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P26490 and previous config saved to /var/cache/conftool/dbconfig/20220425-124401-ladsgroup.json
* 12:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26489 and previous config saved to /var/cache/conftool/dbconfig/20220425-122856-ladsgroup.json
* 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26488 and previous config saved to /var/cache/conftool/dbconfig/20220425-122531-ladsgroup.json
* 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26487 and previous config saved to /var/cache/conftool/dbconfig/20220425-122518-ladsgroup.json
* 12:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26485 and previous config saved to /var/cache/conftool/dbconfig/20220425-121013-ladsgroup.json
* 12:02 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1001.eqiad.wmnet
* 11:58 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-conf1001.eqiad.wmnet
* 11:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P26484 and previous config saved to /var/cache/conftool/dbconfig/20220425-115508-ladsgroup.json
* 11:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26483 and previous config saved to /var/cache/conftool/dbconfig/20220425-114003-ladsgroup.json
* 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26482 and previous config saved to /var/cache/conftool/dbconfig/20220425-113138-ladsgroup.json
* 11:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 11:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26481 and previous config saved to /var/cache/conftool/dbconfig/20220425-113130-ladsgroup.json
* 11:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
* 11:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
* 11:20 moritzm: failover Ganeti master in codfw-test to ganeti-test2003 [[phab:T306499|T306499]]
* 11:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26480 and previous config saved to /var/cache/conftool/dbconfig/20220425-111625-ladsgroup.json
* 11:13 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26479 and previous config saved to /var/cache/conftool/dbconfig/20220425-111315-kormat.json
* 11:11 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host an-master1001.eqiad.wmnet
* 11:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P26478 and previous config saved to /var/cache/conftool/dbconfig/20220425-110119-ladsgroup.json
* 11:01 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1001.eqiad.wmnet
* 10:58 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26477 and previous config saved to /var/cache/conftool/dbconfig/20220425-105811-kormat.json
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26476 and previous config saved to /var/cache/conftool/dbconfig/20220425-104614-ladsgroup.json
* 10:45 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:45 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:43 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26475 and previous config saved to /var/cache/conftool/dbconfig/20220425-104307-kormat.json
* 10:38 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:38 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 10:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:28 kormat@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26474 and previous config saved to /var/cache/conftool/dbconfig/20220425-102803-kormat.json
* 10:24 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:24 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 10:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flowspec1001.eqiad.wmnet
* 10:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host flowspec1001.eqiad.wmnet
* 10:14 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2004.codfw.wmnet
* 10:13 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:07 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2419.codfw.wmnet
* 10:05 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 10:04 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:02 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2419.codfw.wmnet
* 10:02 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2418.codfw.wmnet
* 09:56 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2418.codfw.wmnet
* 09:56 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2417.codfw.wmnet
* 09:55 kormat@cumin1001: dbctl commit (dc=all): 'db1137 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26472 and previous config saved to /var/cache/conftool/dbconfig/20220425-095543-kormat.json
* 09:55 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:55 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1137.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:51 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2417.codfw.wmnet
* 09:50 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2416.codfw.wmnet
* 09:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26471 and previous config saved to /var/cache/conftool/dbconfig/20220425-094600-ladsgroup.json
* 09:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 09:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
* 09:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26470 and previous config saved to /var/cache/conftool/dbconfig/20220425-094552-ladsgroup.json
* 09:43 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2416.codfw.wmnet
* 09:42 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2415.codfw.wmnet
* 09:41 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 09:41 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm2004.codfw.wmnet
* 09:37 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2415.codfw.wmnet
* 09:36 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2414.codfw.wmnet
* 09:31 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2414.codfw.wmnet
* 09:30 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2413.codfw.wmnet
* 09:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26469 and previous config saved to /var/cache/conftool/dbconfig/20220425-093047-ladsgroup.json
* 09:28 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26468 and previous config saved to /var/cache/conftool/dbconfig/20220425-092807-kormat.json
* 09:25 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2413.codfw.wmnet
* 09:23 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mw2412.codfw.wmnet
* 09:16 jelto@cumin1001: START - Cookbook sre.hosts.reboot-single for host mw2412.codfw.wmnet
* 09:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P26467 and previous config saved to /var/cache/conftool/dbconfig/20220425-091542-ladsgroup.json
* 09:13 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26466 and previous config saved to /var/cache/conftool/dbconfig/20220425-091303-kormat.json
* 09:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1155.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 09:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26465 and previous config saved to /var/cache/conftool/dbconfig/20220425-090037-ladsgroup.json
* 08:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1002.eqiad.wmnet
* 08:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1162 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26464 and previous config saved to /var/cache/conftool/dbconfig/20220425-085822-ladsgroup.json
* 08:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 08:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet
* 08:58 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26463 and previous config saved to /var/cache/conftool/dbconfig/20220425-085759-kormat.json
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 08:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 08:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26462 and previous config saved to /var/cache/conftool/dbconfig/20220425-085650-ladsgroup.json
* 08:55 vgutierrez: restart varnish and ats on cp2037 to clear daemon restart alerts
* 08:54 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-master1002.eqiad.wmnet
* 08:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host dumpsdata1007.eqiad.wmnet
* 08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1154.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 9 hosts with reason: Rebooting db1154 [[phab:T303174|T303174]]
* 08:48 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 9 hosts with reason: Rebooting db1154 [[phab:T303174|T303174]]
* 08:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26461 and previous config saved to /var/cache/conftool/dbconfig/20220425-084318-ladsgroup.json
* 08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26460 and previous config saved to /var/cache/conftool/dbconfig/20220425-084256-kormat.json
* 08:42 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26459 and previous config saved to /var/cache/conftool/dbconfig/20220425-084251-kormat.json
* 08:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26458 and previous config saved to /var/cache/conftool/dbconfig/20220425-084145-ladsgroup.json
* 08:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26457 and previous config saved to /var/cache/conftool/dbconfig/20220425-082813-ladsgroup.json
* 08:27 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26456 and previous config saved to /var/cache/conftool/dbconfig/20220425-082747-kormat.json
* 08:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P26455 and previous config saved to /var/cache/conftool/dbconfig/20220425-082640-ladsgroup.json
* 08:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26454 and previous config saved to /var/cache/conftool/dbconfig/20220425-081307-ladsgroup.json
* 08:12 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26453 and previous config saved to /var/cache/conftool/dbconfig/20220425-081244-kormat.json
* 08:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26452 and previous config saved to /var/cache/conftool/dbconfig/20220425-081135-ladsgroup.json
* 08:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26451 and previous config saved to /var/cache/conftool/dbconfig/20220425-080910-ladsgroup.json
* 08:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 08:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 08:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26450 and previous config saved to /var/cache/conftool/dbconfig/20220425-080838-ladsgroup.json
* 07:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader2001.codfw.wmnet
* 07:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26449 and previous config saved to /var/cache/conftool/dbconfig/20220425-075801-ladsgroup.json
* 07:57 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26448 and previous config saved to /var/cache/conftool/dbconfig/20220425-075740-kormat.json
* 07:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26447 and previous config saved to /var/cache/conftool/dbconfig/20220425-075333-ladsgroup.json
* 07:52 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader2001.codfw.wmnet
* 07:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host search-loader1001.eqiad.wmnet
* 07:51 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3315 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26446 and previous config saved to /var/cache/conftool/dbconfig/20220425-075106-kormat.json
* 07:50 kormat@cumin1001: dbctl commit (dc=all): 'db1144:3314 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P26445 and previous config saved to /var/cache/conftool/dbconfig/20220425-075045-kormat.json
* 07:50 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 07:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1144.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 07:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1001.wikimedia.org
* 07:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host search-loader1001.eqiad.wmnet
* 07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26444 and previous config saved to /var/cache/conftool/dbconfig/20220425-074912-ladsgroup.json
* 07:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 07:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 07:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26443 and previous config saved to /var/cache/conftool/dbconfig/20220425-074904-ladsgroup.json
* 07:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host apt1001.wikimedia.org
* 07:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26442 and previous config saved to /var/cache/conftool/dbconfig/20220425-073828-ladsgroup.json
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26441 and previous config saved to /var/cache/conftool/dbconfig/20220425-073359-ladsgroup.json
* 07:31 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host seaborgium.wikimedia.org
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26440 and previous config saved to /var/cache/conftool/dbconfig/20220425-072323-ladsgroup.json
* 07:16 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|4a6c1ddf5cd1a46ab05f5d6fda4b938a3ee37238}}: Remove unnecessary wgNamespaceAliases from bnwiki ([[phab:T318003|T318003]]) (duration: 04m 16s)
* 07:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host seaborgium.wikimedia.org
* 07:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26439 and previous config saved to /var/cache/conftool/dbconfig/20220425-072157-ladsgroup.json
* 07:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 07:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 07:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26438 and previous config saved to /var/cache/conftool/dbconfig/20220425-072149-ladsgroup.json
* 07:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2005.codfw.wmnet
* 07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26437 and previous config saved to /var/cache/conftool/dbconfig/20220425-071853-ladsgroup.json
* 07:15 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2005.codfw.wmnet
* 07:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2004.codfw.wmnet
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:11 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784685{{!}}ActorMigration: Start reading from rev_actor field in group0 (T275246)]] (duration: 00m 50s)
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2004.codfw.wmnet
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26436 and previous config saved to /var/cache/conftool/dbconfig/20220425-070644-ladsgroup.json
* 07:06 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785326{{!}}Add tothemoon.ser.asu.edu to the wgCopyUploadsDomains allowlist of commonswiki (T306671)]] (duration: 00m 52s)
* 07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26435 and previous config saved to /var/cache/conftool/dbconfig/20220425-070348-ladsgroup.json
* 06:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1001.wikimedia.org
* 06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26434 and previous config saved to /var/cache/conftool/dbconfig/20220425-065559-ladsgroup.json
* 06:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 06:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 06:54 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1001.wikimedia.org
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26433 and previous config saved to /var/cache/conftool/dbconfig/20220425-065139-ladsgroup.json
* 06:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1132 into s1 [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P26432 and previous config saved to /var/cache/conftool/dbconfig/20220425-063823-marostegui.json
* 06:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26431 and previous config saved to /var/cache/conftool/dbconfig/20220425-063634-ladsgroup.json
* 06:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26430 and previous config saved to /var/cache/conftool/dbconfig/20220425-063409-ladsgroup.json
* 06:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:11 dcausse: depooling and restarting blazegraph on wdqs1007 (deadlocked for 4+days)
* 04:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26429 and previous config saved to /var/cache/conftool/dbconfig/20220425-045902-ladsgroup.json
* 04:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26428 and previous config saved to /var/cache/conftool/dbconfig/20220425-044357-ladsgroup.json
* 04:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26427 and previous config saved to /var/cache/conftool/dbconfig/20220425-042852-ladsgroup.json
* 04:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26426 and previous config saved to /var/cache/conftool/dbconfig/20220425-041347-ladsgroup.json
* 04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26425 and previous config saved to /var/cache/conftool/dbconfig/20220425-040940-ladsgroup.json
* 04:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 04:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 04:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26424 and previous config saved to /var/cache/conftool/dbconfig/20220425-040926-ladsgroup.json
* 03:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26423 and previous config saved to /var/cache/conftool/dbconfig/20220425-035421-ladsgroup.json
* 03:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26422 and previous config saved to /var/cache/conftool/dbconfig/20220425-033916-ladsgroup.json
* 03:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26421 and previous config saved to /var/cache/conftool/dbconfig/20220425-032410-ladsgroup.json
* 03:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26420 and previous config saved to /var/cache/conftool/dbconfig/20220425-031959-ladsgroup.json
* 03:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 03:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26419 and previous config saved to /var/cache/conftool/dbconfig/20220425-031944-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26418 and previous config saved to /var/cache/conftool/dbconfig/20220425-030439-ladsgroup.json
* 02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26417 and previous config saved to /var/cache/conftool/dbconfig/20220425-024934-ladsgroup.json
* 02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26416 and previous config saved to /var/cache/conftool/dbconfig/20220425-023429-ladsgroup.json
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26415 and previous config saved to /var/cache/conftool/dbconfig/20220425-023020-ladsgroup.json
* 02:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 02:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26414 and previous config saved to /var/cache/conftool/dbconfig/20220425-023012-ladsgroup.json
* 02:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26413 and previous config saved to /var/cache/conftool/dbconfig/20220425-021507-ladsgroup.json
* 02:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26412 and previous config saved to /var/cache/conftool/dbconfig/20220425-020002-ladsgroup.json
* 01:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26411 and previous config saved to /var/cache/conftool/dbconfig/20220425-014457-ladsgroup.json
* 01:40 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: update new codfw1dev host (duration: 00m 54s)
* 01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
* 01:39 andrew@deploy1002: Started deploy [horizon/deploy@9d02cd6]: update new codfw1dev host
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26410 and previous config saved to /var/cache/conftool/dbconfig/20220425-010952-ladsgroup.json
* 01:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 01:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26409 and previous config saved to /var/cache/conftool/dbconfig/20220425-010938-ladsgroup.json
* 00:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26408 and previous config saved to /var/cache/conftool/dbconfig/20220425-005432-ladsgroup.json
* 00:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26407 and previous config saved to /var/cache/conftool/dbconfig/20220425-003927-ladsgroup.json
* 00:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26406 and previous config saved to /var/cache/conftool/dbconfig/20220425-002422-ladsgroup.json
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26405 and previous config saved to /var/cache/conftool/dbconfig/20220425-001152-ladsgroup.json
* 00:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 00:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 00:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26404 and previous config saved to /var/cache/conftool/dbconfig/20220425-001144-ladsgroup.json


== 2022-04-24 ==
== 2022-09-17 ==
* 23:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26403 and previous config saved to /var/cache/conftool/dbconfig/20220424-235639-ladsgroup.json
* 12:17 Emperor: set thanos ring replicas to 3.80 [[phab:T311690|T311690]]
* 23:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26402 and previous config saved to /var/cache/conftool/dbconfig/20220424-234134-ladsgroup.json
* 10:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34879 and previous config saved to /var/cache/conftool/dbconfig/20220917-103903-ladsgroup.json
* 23:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26401 and previous config saved to /var/cache/conftool/dbconfig/20220424-232629-ladsgroup.json
* 10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P34878 and previous config saved to /var/cache/conftool/dbconfig/20220917-102356-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26400 and previous config saved to /var/cache/conftool/dbconfig/20220424-232219-ladsgroup.json
* 10:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P34877 and previous config saved to /var/cache/conftool/dbconfig/20220917-100850-ladsgroup.json
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 09:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2105 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34876 and previous config saved to /var/cache/conftool/dbconfig/20220917-095344-ladsgroup.json
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34875 and previous config saved to /var/cache/conftool/dbconfig/20220917-094856-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26399 and previous config saved to /var/cache/conftool/dbconfig/20220424-232205-ladsgroup.json
* 09:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P34874 and previous config saved to /var/cache/conftool/dbconfig/20220917-093349-ladsgroup.json
* 23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26398 and previous config saved to /var/cache/conftool/dbconfig/20220424-230700-ladsgroup.json
* 09:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P34873 and previous config saved to /var/cache/conftool/dbconfig/20220917-091843-ladsgroup.json
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26397 and previous config saved to /var/cache/conftool/dbconfig/20220424-230136-ladsgroup.json
* 09:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34872 and previous config saved to /var/cache/conftool/dbconfig/20220917-090336-ladsgroup.json
* 22:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26396 and previous config saved to /var/cache/conftool/dbconfig/20220424-225155-ladsgroup.json
* 07:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34871 and previous config saved to /var/cache/conftool/dbconfig/20220917-074806-ladsgroup.json
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26395 and previous config saved to /var/cache/conftool/dbconfig/20220424-224631-ladsgroup.json
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P34870 and previous config saved to /var/cache/conftool/dbconfig/20220917-073300-ladsgroup.json
* 22:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26394 and previous config saved to /var/cache/conftool/dbconfig/20220424-223650-ladsgroup.json
* 07:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P34869 and previous config saved to /var/cache/conftool/dbconfig/20220917-071753-ladsgroup.json
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26393 and previous config saved to /var/cache/conftool/dbconfig/20220424-223240-ladsgroup.json
* 07:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2129 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34868 and previous config saved to /var/cache/conftool/dbconfig/20220917-070247-ladsgroup.json
* 22:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 05:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2105 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34867 and previous config saved to /var/cache/conftool/dbconfig/20220917-051719-ladsgroup.json
* 22:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 05:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26392 and previous config saved to /var/cache/conftool/dbconfig/20220424-223232-ladsgroup.json
* 05:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P26391 and previous config saved to /var/cache/conftool/dbconfig/20220424-223126-ladsgroup.json
* 05:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2129 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34866 and previous config saved to /var/cache/conftool/dbconfig/20220917-051527-ladsgroup.json
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26390 and previous config saved to /var/cache/conftool/dbconfig/20220424-221727-ladsgroup.json
* 05:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 22:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26389 and previous config saved to /var/cache/conftool/dbconfig/20220424-221621-ladsgroup.json
* 05:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 22:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26388 and previous config saved to /var/cache/conftool/dbconfig/20220424-220222-ladsgroup.json
* 05:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34865 and previous config saved to /var/cache/conftool/dbconfig/20220917-051203-ladsgroup.json
* 21:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26387 and previous config saved to /var/cache/conftool/dbconfig/20220424-214717-ladsgroup.json
* 05:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26386 and previous config saved to /var/cache/conftool/dbconfig/20220424-213440-ladsgroup.json
* 05:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 21:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26385 and previous config saved to /var/cache/conftool/dbconfig/20220424-213425-ladsgroup.json
* 21:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26384 and previous config saved to /var/cache/conftool/dbconfig/20220424-211920-ladsgroup.json
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26383 and previous config saved to /var/cache/conftool/dbconfig/20220424-211607-ladsgroup.json
* 21:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26382 and previous config saved to /var/cache/conftool/dbconfig/20220424-211559-ladsgroup.json
* 21:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26381 and previous config saved to /var/cache/conftool/dbconfig/20220424-210521-ladsgroup.json
* 21:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26380 and previous config saved to /var/cache/conftool/dbconfig/20220424-210415-ladsgroup.json
* 21:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26379 and previous config saved to /var/cache/conftool/dbconfig/20220424-210052-ladsgroup.json
* 20:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26378 and previous config saved to /var/cache/conftool/dbconfig/20220424-205016-ladsgroup.json
* 20:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26377 and previous config saved to /var/cache/conftool/dbconfig/20220424-204910-ladsgroup.json
* 20:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P26376 and previous config saved to /var/cache/conftool/dbconfig/20220424-204547-ladsgroup.json
* 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26375 and previous config saved to /var/cache/conftool/dbconfig/20220424-203639-ladsgroup.json
* 20:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 20:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26374 and previous config saved to /var/cache/conftool/dbconfig/20220424-203630-ladsgroup.json
* 20:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P26373 and previous config saved to /var/cache/conftool/dbconfig/20220424-203511-ladsgroup.json
* 20:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26372 and previous config saved to /var/cache/conftool/dbconfig/20220424-203042-ladsgroup.json
* 20:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26371 and previous config saved to /var/cache/conftool/dbconfig/20220424-202125-ladsgroup.json
* 20:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26370 and previous config saved to /var/cache/conftool/dbconfig/20220424-202006-ladsgroup.json
* 20:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26369 and previous config saved to /var/cache/conftool/dbconfig/20220424-200620-ladsgroup.json
* 19:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26368 and previous config saved to /var/cache/conftool/dbconfig/20220424-195115-ladsgroup.json
* 19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26367 and previous config saved to /var/cache/conftool/dbconfig/20220424-194705-ladsgroup.json
* 19:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 19:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26366 and previous config saved to /var/cache/conftool/dbconfig/20220424-194651-ladsgroup.json
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1127 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26365 and previous config saved to /var/cache/conftool/dbconfig/20220424-193635-ladsgroup.json
* 19:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
* 19:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26364 and previous config saved to /var/cache/conftool/dbconfig/20220424-193611-ladsgroup.json
* 19:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26363 and previous config saved to /var/cache/conftool/dbconfig/20220424-193146-ladsgroup.json
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26362 and previous config saved to /var/cache/conftool/dbconfig/20220424-193028-ladsgroup.json
* 19:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 19:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 19:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26361 and previous config saved to /var/cache/conftool/dbconfig/20220424-193020-ladsgroup.json
* 19:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26360 and previous config saved to /var/cache/conftool/dbconfig/20220424-192106-ladsgroup.json
* 19:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P26359 and previous config saved to /var/cache/conftool/dbconfig/20220424-191641-ladsgroup.json
* 19:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26358 and previous config saved to /var/cache/conftool/dbconfig/20220424-191515-ladsgroup.json
* 19:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26357 and previous config saved to /var/cache/conftool/dbconfig/20220424-190601-ladsgroup.json
* 19:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26356 and previous config saved to /var/cache/conftool/dbconfig/20220424-190135-ladsgroup.json
* 19:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P26355 and previous config saved to /var/cache/conftool/dbconfig/20220424-190008-ladsgroup.json
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26354 and previous config saved to /var/cache/conftool/dbconfig/20220424-185724-ladsgroup.json
* 18:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 18:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26353 and previous config saved to /var/cache/conftool/dbconfig/20220424-185717-ladsgroup.json
* 18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26352 and previous config saved to /var/cache/conftool/dbconfig/20220424-185056-ladsgroup.json
* 18:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26351 and previous config saved to /var/cache/conftool/dbconfig/20220424-184503-ladsgroup.json
* 18:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26350 and previous config saved to /var/cache/conftool/dbconfig/20220424-184212-ladsgroup.json
* 18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26349 and previous config saved to /var/cache/conftool/dbconfig/20220424-183813-ladsgroup.json
* 18:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 18:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26348 and previous config saved to /var/cache/conftool/dbconfig/20220424-183805-ladsgroup.json
* 18:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26347 and previous config saved to /var/cache/conftool/dbconfig/20220424-182707-ladsgroup.json
* 18:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26346 and previous config saved to /var/cache/conftool/dbconfig/20220424-182300-ladsgroup.json
* 18:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26345 and previous config saved to /var/cache/conftool/dbconfig/20220424-181201-ladsgroup.json
* 18:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P26344 and previous config saved to /var/cache/conftool/dbconfig/20220424-180755-ladsgroup.json
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26343 and previous config saved to /var/cache/conftool/dbconfig/20220424-180555-ladsgroup.json
* 18:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26342 and previous config saved to /var/cache/conftool/dbconfig/20220424-180530-ladsgroup.json
* 18:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26341 and previous config saved to /var/cache/conftool/dbconfig/20220424-180013-ladsgroup.json
* 18:00 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 18:00 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 17:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26340 and previous config saved to /var/cache/conftool/dbconfig/20220424-175250-ladsgroup.json
* 17:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26339 and previous config saved to /var/cache/conftool/dbconfig/20220424-175025-ladsgroup.json
* 17:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26338 and previous config saved to /var/cache/conftool/dbconfig/20220424-174553-ladsgroup.json
* 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 17:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 17:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26337 and previous config saved to /var/cache/conftool/dbconfig/20220424-173955-ladsgroup.json
* 17:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P26336 and previous config saved to /var/cache/conftool/dbconfig/20220424-173520-ladsgroup.json
* 17:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26335 and previous config saved to /var/cache/conftool/dbconfig/20220424-172450-ladsgroup.json
* 17:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26334 and previous config saved to /var/cache/conftool/dbconfig/20220424-172015-ladsgroup.json
* 17:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P26333 and previous config saved to /var/cache/conftool/dbconfig/20220424-170945-ladsgroup.json
* 16:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26332 and previous config saved to /var/cache/conftool/dbconfig/20220424-165439-ladsgroup.json
* 16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26331 and previous config saved to /var/cache/conftool/dbconfig/20220424-164748-ladsgroup.json
* 16:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 16:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 16:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 16:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26330 and previous config saved to /var/cache/conftool/dbconfig/20220424-163151-ladsgroup.json
* 16:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 16:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 16:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 16:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 03:20 Amir1: optimizing flaggedtemplates on plwiki (s2) in db2088
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
* 01:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26329 and previous config saved to /var/cache/conftool/dbconfig/20220424-012830-ladsgroup.json
* 01:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26328 and previous config saved to /var/cache/conftool/dbconfig/20220424-011325-ladsgroup.json
* 00:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P26327 and previous config saved to /var/cache/conftool/dbconfig/20220424-005820-ladsgroup.json
* 00:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26326 and previous config saved to /var/cache/conftool/dbconfig/20220424-004315-ladsgroup.json


== 2022-04-23 ==
== 2022-09-16 ==
* 23:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26325 and previous config saved to /var/cache/conftool/dbconfig/20220423-232748-ladsgroup.json
* 21:29 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 21:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 23:27 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,
* 21:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34864 and previous config saved to /


== 2022-04-22 ==
== 2022-09-15 ==
* 23:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P26233 and previous config saved to /var/cache/conftool/dbconfig/20220422-235334-ladsgroup.json
* 23:51 mutante: gerrit1001 - disabled puppet - gerrit:832411
* 23:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26232 and previous config saved to /var/cache/conftool/dbconfig/20220422-233829-ladsgroup.json
* 22:01 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wcqs2001.codfw.wmnet with reason: [[phab:T316236|T316236]]
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26231 and previous config saved to /var/cache/conftool/dbconfig/20220422-232210-ladsgroup.json
* 22:01 bking@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wcqs2001.codfw.wmnet with reason: [[phab:T316236|T316236]]
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0
* 21:30 ebernhardson: depool wcqs2001 for [[phab:T316236|T316236]]
* 20:25 thcipriani@deploy1002: Finished scap: Backport for [[gerrit:832526{{!}}Increase coverage of Research Incentive Survey on idwiki (T316466)]] (duration: 07m 06s)
* 20:18 thcipriani@deploy1002: thcipriani


== 2022-04-21 ==
== 2022-09-14 ==
* 23:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26111 and previous config saved to /var/cache/conftool/dbconfig/20220421-235814-ladsgroup.json
* 22:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34739 and previous config saved to /var/cache/conftool/dbconfig/20220914-220822-ladsgroup.json
* 23:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 22:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 23:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 22:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 23:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26110 and previous config saved to /var/cache/conftool/dbconfig/20220421-235203-ladsgroup.json
* 22:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P26109 and previous config saved to /var/cache/conftool/dbconfig/20220421-233658-ladsgroup.json
* 22:07 ladsgroup@cumin1001: START - Cookbook sre
* 23:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T306560|T306560]])', diff saved to https://phabricator.wikimedia.org/P26108 and previous config saved to /var/cache/conftool/dbconfig/20220421-233212-ladsgroup.json
* 23:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 23:25 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 23:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1136 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P26107 and previous config saved to /var/cache
* 08:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 08:30 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 08:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 08:30 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 08:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 08:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 08:29 jynus@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1005.eqiad.wmnet with reason: host reimage
* 08:25 ladsgroup@deploy1002: ladsgroup and ladsgroup: Backport for [[gerrit:832157{{!}}Stop writing to the old templatelinks columns of enwiki (T312865)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 08:25 jynus@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1005.eqiad.wmnet with reason: host reimage
* 08:25 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:832157{{!}}Stop writing to the old templatelinks columns of enwiki (T312865)]]
* 08:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25918 and previous config saved to /var/cache/conftool/dbconfig/20220421-082249-ladsgroup.json
* 08:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 08:19 ladsgroup@cumin1001: dbctl commit
* 08:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 08:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 08:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 08:03 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es1024.eqiad.wmnet with reason: down
* 08:03 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on es1024.eqiad.wmnet with reason: down
* 08:02 marostegui@deploy1002: Synchronized wmf-config/db-production.php: Enable writes on es5 [[phab:T317739|T317739]] (duration: 03m 38s)
* 07:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 07:57 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1024 [[phab:T317739|T317739]]', diff saved to https://phabricator.wikimedia.org/P34703 and previous config saved to /var/cache/conftool/dbconfig/20220914-075722-root.json
* 07:55 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1023 to es5 primary [[phab:T317739|T317739]]', diff saved to https://phabricator.wikimedia.org/P34702 and previous config saved to /var/cache/conftool/dbconfig/20220914-075550-marostegui.json
* 07:55 marostegui: Starting es5 eqiad failover from es1024 to es1023 [[phab:T317739|T317739]]
* 07:54 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 07:54 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 07:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:50 marostegui@deploy1002: Synchronized wmf-config/db-production.php: Disable writes on es5 [[phab:T317739|T317739]] (duration: 04m 13s)
* 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Set es1023 with weight 0 [[phab:T317739|T317739]]', diff saved to https://phabricator.wikimedia.org/P34701 and previous config saved to /var/cache/conftool/dbconfig/20220914-074617-marostegui.json
* 07:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 [[phab:T317739|T317739]]
* 07:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 [[phab:T317739|T317739]]
* 07:42 marostegui@cumin1001: dbctl commit (dc=all): 'db2123 (re)pooling @ 100%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34700 and previous config saved to /var/cache/conftool/dbconfig/20220914-074248-root.json
* 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'db2123 (re)pooling @ 75%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34699 and previous config saved to /var/cache/conftool/dbconfig/20220914-072743-root.json
* 07:12 marostegui@cumin1001: dbctl commit (dc=


== 2022-04-20 ==
== 2022-09-09 ==
* 23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25833 and previous config saved to /var/cache/conftool/dbconfig/20220420-234831-ladsgroup.json
* 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34353 and previous config saved to /var/cache/conftool/dbconfig/20220909-224245-ladsgroup.json
* 23:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 22:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 23:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 22:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance
* 23:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 22:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34352 and previous config saved to /var/cache/conftool/dbconfig/20220909-224223-ladsgroup.json
* 23:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 22:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P34351 and previous config saved to /var/cache/conftool/dbconfig/20220909-222717-ladsgroup.json
* 23:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25832 and previous config saved to /var/cache/conftool/dbconfig/20220420-234818-ladsgroup.json
* 22:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P34350 and previous config saved to /var/cache/conftool/dbconfig/20220909-221210-ladsgroup.json
* 23:36 mutante: kubernetes/puppetmaster: added deployment/user tokens for new service image-suggestion [[phab:T304891|T304891]]
* 21:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34349 and previous config saved to /var/cache/conftool/dbconfig/20220909-215704-ladsgroup.json
* 23:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 20:27 herron@cumin1001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=93) for new host dispatch-be1001.eqiad.wmnet
* 23:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 20:27 herron@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dispatch-be1001.eqiad.wmnet on all recursors
* 23:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25831 and previous config saved to /var/cache/conftool/dbconfig/20220420-233313-ladsgroup.json
* 20:27 herron@cumin1001: START - Cookbook sre.dns.wipe-cache dispatch-be1001.eqiad.wmnet on all recursors
* 23:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25830 and previous config saved to /var/cache/conftool/dbconfig/20220420-231808-ladsgroup.json
* 20:27 herron@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 23:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25829 and previous config saved to /var/cache/conftool/dbconfig/20220420-231645-ladsgroup.json
* 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2116 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34348 and previous config saved to /var/cache/conftool/dbconfig/20220909-201857-ladsgroup.json
* 23:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25828 and previous config saved to /var/cache/conftool/dbconfig/20220420-230303-ladsgroup.json
* 20:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
* 23:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25827 and previous config saved to /var/cache/conftool/dbconfig/20220420-230140-ladsgroup.json
* 20:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
* 22:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25826 and previous config saved to /var/cache/conftool/dbconfig/20220420-225643-ladsgroup.json
* 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2112 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34347 and previous config saved to /var/cache/conftool/dbconfig/20220909-201835-ladsgroup.json
* 22:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 20:03 herron@cumin1001: START - Cookbook sre.dns.netbox
* 22:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 20:03 herron@cumin1001: START - Cookbook sre.ganeti.makevm for new host dispatch-be1001.eqiad.wmnet
* 22:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P34345 and previous config saved to /var/cache/conftool/dbconfig/20220909-200329-ladsgroup.json
* 22:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 20:02 herron@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons.
* 22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 19:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P34344 and previous config saved to /var/cache/conftool/dbconfig/20220909-194822-ladsgroup.json
* 22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 19:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 19:45 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 19:45 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25825 and previous config saved to /var/cache/conftool/dbconfig/20220420-224634-ladsgroup.json
* 19:44 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2112 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34343 and previous config saved to /var/cache/conftool/dbconfig/20220909-193316-ladsgroup.json
* 22:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 18:21 herron@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons.
* 22:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25824 and previous config saved to /var/cache/conftool/dbconfig/20220420-223129-ladsgroup.json
* 16:21 herron@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons.
* 22:14 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS buster
* 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34342 and previous config saved to /var/cache/conftool/dbconfig/20220909-155234-ladsgroup.json
* 22:13 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet with OS buster
* 15:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
* 22:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25823 and previous config saved to /var/cache/conftool/dbconfig/20220420-220048-ladsgroup.json
* 15:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25822 and previous config saved to /var/cache/conftool/dbconfig/20220420-215818-ladsgroup.json
* 15:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34341 and previous config saved to /var/cache/conftool/dbconfig/20220909-155213-ladsgroup.json
* 21:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P34340 and previous config saved to /var/cache/conftool/dbconfig/20220909-153706-ladsgroup.json
* 21:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 15:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P34339 and previous config saved to /var/cache/conftool/dbconfig/20220909-152159-ladsgroup.json
* 21:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25821 and previous config saved to /var/cache/conftool/dbconfig/20220420-215810-ladsgroup.json
* 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34338 and previous config saved to /var/cache/conftool/dbconfig/20220909-150651-ladsgroup.json
* 21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25820 and previous config saved to /var/cache/conftool/dbconfig/20220420-214305-ladsgroup.json
* 14:44 moritzm: imported jenkins 2.346.3 to thirdparty/ci
* 21:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:43 dcausse@deploy1002: Synchronized wmf-config/InitialiseSettings.php: [[phab:T317381|T317381]]: Revert "Disable CirrusSearch completion suggester" (duration: 03m 57s)
* 21:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:40 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 14:40 herron@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons.
* 21:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 14:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 14:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:33 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 14:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2112 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34336 and previous config saved to /var/cache/conftool/dbconfig/20220909-133846-ladsgroup.json
* 21:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: Maintenance
* 21:32 jhuneidi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784711{{!}}Revert "Revert "Create 'uploader' group for viwiki""]] (duration: 00m 53s)
* 13:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: Maintenance
* 21:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25819 and previous config saved to /var/cache/conftool/dbconfig/20220420-213115-ladsgroup.json
* 13:33 dcausse: restartin blazegraph on wdqs2003 (BlazegraphFreeAllocatorsDecreasingRapidly)
* 21:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 13:03 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 21:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:24 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 21:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25818 and previous config saved to /var/cache/conftool/dbconfig/20220420-212800-ladsgroup.json
* 12:24 ayounsi@cumin1001: START - Cookbook sre.network.cf
* 21:27 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 12:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34334 and previous config saved to /var/cache/conftool/dbconfig/20220909-120029-ladsgroup.json
* 21:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 11:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P34333 and previous config saved to /var/cache/conftool/dbconfig/20220909-114522-ladsgroup.json
* 21:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P34331 and previous config saved to /var/cache/conftool/dbconfig/20220909-113016-ladsgroup.json
* 21:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 11:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34330 and previous config saved to /var/cache/conftool/dbconfig/20220909-111509-ladsgroup.json
* 21:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25817 and previous config saved to /var/cache/conftool/dbconfig/20220420-211255-ladsgroup.json
* 10:33 marostegui@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34329 and previous config saved to /var/cache/conftool/dbconfig/20220909-103334-root.json
* 21:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 10:31 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s)
* 21:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 10:31 bmansurov@deploy1002: Started deploy [airflow-dags/research@b9be20d]: (no justification provided)
* 21:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 10:18 marostegui@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34328 and previous config saved to /var/cache/conftool/dbconfig/20220909-101830-root.json
* 21:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 10:03 marostegui@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34327 and previous config saved to /var/cache/conftool/dbconfig/20220909-100324-root.json
* 21:07 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 09:53 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s)
* 21:05 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
* 09:53 bmansurov@deploy1002: Started deploy [airflow-dags/research@b9be20d]: (no justification provided)
* 09:48 marostegui@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34326 and previous config saved to /var/cache/conftool/dbconfig/20220909-094819-root.json
* 09:33 marostegui@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34325 and previous config saved to /var/cache/conftool/dbconfig/20220909-093314-root.json
* 09:18 marostegui@cumin1001: dbctl commit (dc=all): 'db1137 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34323 and previous config saved to /var/cache/conftool/dbconfig/20220909-091809-root.json
* 08:59 cgoubert@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wtp[1029-1033].eqiad.wmnet
* 08:59 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:56 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 08:37 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp[1029-1033].eqiad.wmnet
* 08:32 dcausse: rebuilding all completion indices in elastic@codfw
* 08:16 dcausse: restarting on blazegraph on wdqs2002 (BlazegraphFreeAllocatorsDecreasingRapidly)
* 08:12 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2106 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34322 and previous config saved to /var/cache/conftool/dbconfig/20220909-081103-ladsgroup.json
* 08:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
* 08:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance
* 08:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P34321 and previous config saved to /var/cache/conftool/dbconfig/20220909-081042-ladsgroup.json
* 08:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 08:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 08:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
* 08:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2097.codfw.wmnet with reason: Maintenance
* 08:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34320 and previous config saved to /var/cache/conftool/dbconfig/20220909-080609-ladsgroup.json
* 08:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 08:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 08:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 08:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 07:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P34319 and previous config saved to /var/cache/conftool/dbconfig/20220909-074710-ladsgroup.json
* 07:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 07:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 07:46 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 07:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet
* 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1137 for upgrade', diff saved to https://phabricator.wikimedia.org/P34318 and previous config saved to /var/cache/conftool/dbconfig/20220909-071255-root.json
* 07:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet
* 05:44 marostegui: dbmaint s8 wikidatawiki eqiad [[phab:T317349|T317349]]
* 05:43 marostegui: dbmaint s3 testwikidatawiki eqiad [[phab:T317349|T317349]]
* 05:42 marostegui: dbmaint s4 commonswiki eqiad [[phab:T317349|T317349]]
* 05:41 marostegui: dbmaint s4 testcommonswiki eqiad [[phab:T317349|T317349]]
* 05:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 05:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 05:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 05:11 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: cirrus: Switch all wikis from completion suggester to prefix search, yesterdays completion index builds in codfw weren't all succesfull and users are getting incomplete results (duration: 04m 01s)
* 05:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 00:09 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 22s)
* 00:08 bmansurov@deploy1002: Started deploy [airflow-dags/research@b9be20d]: (no justification provided)
 
== 2022-09-08 ==
* 23:56 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 27s)
* 23:55 bmansurov@deploy1002: Started deploy [airflow-dags/research@b9be20d]: (no justification provided)
* 21:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:08 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.28  refs [[phab:T314189|T314189]]
* 21:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:02 TheresNoTime: closing UTC late backport and config training
* 21:01 samtar@deploy1002: Finished scap: Backport for [[gerrit:830703{{!}}Fix selser on html endpoints (T317215)]] (duration: 06m 48s)
* 20:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:55 samtar@deploy1002: samtar and arlolra: Backport for [[gerrit:830703{{!}}Fix selser on html endpoints (T317215)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 20:55 samtar@deploy1002: Started scap: Backport for [[gerrit:830703{{!}}Fix selser on html endpoints (T317215)]]
* 20:49 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:37 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:33 samtar@deploy1002: Finished scap: Backport for [[gerrit:830702{{!}}Fix selser on html endpoints (T317215)]] (duration: 12m 06s)
* 20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:21 samtar@deploy1002: samtar and arlolra: Backport for [[gerrit:830702{{!}}Fix selser on html endpoints (T317215)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 20:21 samtar@deploy1002: Started scap: Backport for [[gerrit:830702{{!}}Fix selser on html endpoints (T317215)]]
* 20:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:56 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.39.0-wmf.27  refs [[phab:T314189|T314189]]
* 19:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:36 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.28  refs [[phab:T314189|T314189]]
* 19:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:15 jhuneidi@deploy1002: Synchronized php: group1 wikis to 1.39.0-wmf.28  refs [[phab:T314189|T314189]] (duration: 03m 39s)
* 19:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:11 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.28  refs [[phab:T314189|T314189]]
* 17:33 sukhe: stat1008: sudo ipmitool -I lanplus -H "stat1008.mgmt.eqiad.wmnet" -U root -E chassis power cycle
* 17:22 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 17:22 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 17:22 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 17:21 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 17:21 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 17:20 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 16:22 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host kafka-logging1005.mgmt.eqiad.wmnet with reboot policy FORCED
* 16:04 dancy@deploy1002: Installing scap version "4.17.0" for 566 hosts
* 15:57 pt1979@cumin1001: START - Cookbook sre.hosts.provision for host kafka-logging1005.mgmt.eqiad.wmnet with reboot policy FORCED
* 15:55 pt1979@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:52 pt1979@cumin1001: START - Cookbook sre.dns.netbox
* 15:51 pt1979@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:50 cgoubert@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid
* 15:49 pt1979@cumin1001: START - Cookbook sre.dns.netbox
* 15:45 cgoubert@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:830803{{!}}wtp: Purge wtp servers following migration to parse (T317025)]] (duration: 04m 00s)
* 15:40 pt1979@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:39 pt1979@cumin1001: START - Cookbook sre.dns.netbox
* 15:36 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=appserver
* 15:36 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=api_appserver
* 15:35 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=api-https
* 15:33 akosiaris: restart etcdmirror on conf2005
* 15:28 cgoubert@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:830803{{!}}wtp: Purge wtp servers following migration to parse (T317025)]] (duration: 12m 48s)
* 15:25 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
* 15:25 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: sync on main
* 15:25 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
* 15:24 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: sync on main
* 15:21 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
* 15:21 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: sync on main
* 15:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 15:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 15:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 15:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 15:11 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 15:02 moritzm: installing nginx security updates on bullseye
* 14:58 papaul: maintenance on mr1-codfw complete
* 14:57 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp[1025-1028,1048].eqiad.wmnet
* 14:57 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:54 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 14:40 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp[1025-1028,1048].eqiad.wmnet
* 14:38 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp[1043-1047].eqiad.wmnet
* 14:38 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:38 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 14:36 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 14:35 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 14:25 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp[1043-1047].eqiad.wmnet
* 14:23 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp[1038-1042].eqiad.wmnet
* 14:23 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:20 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 14:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: Will do maint later', diff saved to https://phabricator.wikimedia.org/P34312 and previous config saved to /var/cache/conftool/dbconfig/20220908-142029-ladsgroup.json
* 14:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: Maint over long time ago', diff saved to https://phabricator.wikimedia.org/P34311 and previous config saved to /var/cache/conftool/dbconfig/20220908-141600-ladsgroup.json
* 14:07 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp[1038-1042].eqiad.wmnet
* 14:06 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp1037.eqiad.wmnet
* 14:06 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: Will do maint later', diff saved to https://phabricator.wikimedia.org/P34310 and previous config saved to /var/cache/conftool/dbconfig/20220908-140524-ladsgroup.json
* 14:04 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 14:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 75%: Maint over long time ago', diff saved to https://phabricator.wikimedia.org/P34309 and previous config saved to /var/cache/conftool/dbconfig/20220908-140055-ladsgroup.json
* 14:00 papaul: on going maintenance on mr1-codfw
* 13:57 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp1037.eqiad.wmnet
* 13:55 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp1036.eqiad.wmnet
* 13:55 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:53 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 13:51 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 13:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Will do maint later', diff saved to https://phabricator.wikimedia.org/P34307 and previous config saved to /var/cache/conftool/dbconfig/20220908-135019-ladsgroup.json
* 13:47 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp1036.eqiad.wmnet
* 13:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 25%: Maint over long time ago', diff saved to https://phabricator.wikimedia.org/P34305 and previous config saved to /var/cache/conftool/dbconfig/20220908-134550-ladsgroup.json
* 13:43 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp1035.eqiad.wmnet
* 13:43 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:43 vgutierrez: rolling upgrade to ats 9 in cp drmrs - [[phab:T309651|T309651]]
* 13:41 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 13:39 vgutierrez: disable puppet on A:cp-drmrs during the update to ATS 9.1.3 - [[phab:T309651|T309651]]
* 13:36 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp1035.eqiad.wmnet
* 13:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1157 (re)pooling @ 10%: Will do maint later', diff saved to https://phabricator.wikimedia.org/P34304 and previous config saved to /var/cache/conftool/dbconfig/20220908-133514-ladsgroup.json
* 13:31 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wtp1034.eqiad.wmnet
* 13:31 cgoubert@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 10%: Maint over long time ago', diff saved to https://phabricator.wikimedia.org/P34303 and previous config saved to /var/cache/conftool/dbconfig/20220908-133045-ladsgroup.json
* 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34302 and previous config saved to /var/cache/conftool/dbconfig/20220908-133036-root.json
* 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34301 and previous config saved to /var/cache/conftool/dbconfig/20220908-133031-root.json
* 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34300 and previous config saved to /var/cache/conftool/dbconfig/20220908-133024-root.json
* 13:29 moritzm: installing apache2 security updates on Bullseye
* 13:28 cgoubert@cumin1001: START - Cookbook sre.dns.netbox
* 13:23 cgoubert@cumin1001: START - Cookbook sre.hosts.decommission for hosts wtp1034.eqiad.wmnet
* 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34299 and previous config saved to /var/cache/conftool/dbconfig/20220908-131531-root.json
* 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34298 and previous config saved to /var/cache/conftool/dbconfig/20220908-131526-root.json
* 13:15 marostegui@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34297 and previous config saved to /var/cache/conftool/dbconfig/20220908-131519-root.json
* 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34296 and previous config saved to /var/cache/conftool/dbconfig/20220908-130026-root.json
* 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34295 and previous config saved to /var/cache/conftool/dbconfig/20220908-130021-root.json
* 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34294 and previous config saved to /var/cache/conftool/dbconfig/20220908-130014-root.json
* 12:56 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@9e4ed94]: (no justification provided) (duration: 00m 09s)
* 12:56 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@9e4ed94]: (no justification provided)
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34293 and previous config saved to /var/cache/conftool/dbconfig/20220908-124521-root.json
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34292 and previous config saved to /var/cache/conftool/dbconfig/20220908-124516-root.json
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34291 and previous config saved to /var/cache/conftool/dbconfig/20220908-124509-root.json
* 12:42 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@9e4ed94]: (no justification provided) (duration: 00m 09s)
* 12:42 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@9e4ed94]: (no justification provided)
* 12:39 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es1027 to es1 eqiad master, promote es1026 to es2 eqiad master, promote es1028 to es3 eqiad master', diff saved to https://phabricator.wikimedia.org/P34290 and previous config saved to /var/cache/conftool/dbconfig/20220908-123955-marostegui.json
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34289 and previous config saved to /var/cache/conftool/dbconfig/20220908-123016-root.json
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34288 and previous config saved to /var/cache/conftool/dbconfig/20220908-123011-root.json
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34287 and previous config saved to /var/cache/conftool/dbconfig/20220908-123004-root.json
* 12:26 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1033.eqiad.wmnet
* 12:26 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1032.eqiad.wmnet
* 12:26 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wtp[1032-1033].eqiad.wmnet with reason: Downtiming replaced wtp servers
* 12:26 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wtp[1032-1033].eqiad.wmnet with reason: Downtiming replaced wtp servers
* 12:25 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wtp[1032-1033].mgmt with reason: Downtiming replaced wtp servers
* 12:25 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wtp[1032-1033].mgmt with reason: Downtiming replaced wtp servers
* 12:17 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 12:15 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'es1031 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34286 and previous config saved to /var/cache/conftool/dbconfig/20220908-121511-root.json
* 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'es1030 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34285 and previous config saved to /var/cache/conftool/dbconfig/20220908-121506-root.json
* 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'es1029 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34284 and previous config saved to /var/cache/conftool/dbconfig/20220908-121459-root.json
* 12:12 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 12:11 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 12:09 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 12:06 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 12:05 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1029 es1030 es1031 for upgrade', diff saved to https://phabricator.wikimedia.org/P34283 and previous config saved to /var/cache/conftool/dbconfig/20220908-120528-root.json
* 12:04 marostegui@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34282 and previous config saved to /var/cache/conftool/dbconfig/20220908-120439-root.json
* 12:04 marostegui@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34281 and previous config saved to /var/cache/conftool/dbconfig/20220908-120435-root.json
* 12:04 marostegui@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34280 and previous config saved to /var/cache/conftool/dbconfig/20220908-120427-root.json
* 12:03 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'es2025 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34278 and previous config saved to /var/cache/conftool/dbconfig/20220908-115407-root.json
* 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'es2022 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34277 and previous config saved to /var/cache/conftool/dbconfig/20220908-115401-root.json
* 11:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34276 and previous config saved to /var/cache/conftool/dbconfig/20220908-115355-root.json
* 11:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34275 and previous config saved to /var/cache/conftool/dbconfig/20220908-115351-root.json
* 11:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34274 and previous config saved to /var/cache/conftool/dbconfig/20220908-114934-root.json
* 11:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34273 and previous config saved to /var/cache/conftool/dbconfig/20220908-114930-root.json
* 11:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34272 and previous config saved to /var/cache/conftool/dbconfig/20220908-114922-root.json
* 11:39 marostegui@cumin1001: dbctl commit (dc=all): 'es2025 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34271 and previous config saved to /var/cache/conftool/dbconfig/20220908-113902-root.json
* 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'es2022 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34270 and previous config saved to /var/cache/conftool/dbconfig/20220908-113856-root.json
* 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34269 and previous config saved to /var/cache/conftool/dbconfig/20220908-113850-root.json
* 11:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34268 and previous config saved to /var/cache/conftool/dbconfig/20220908-113846-root.json
* 11:34 marostegui@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34267 and previous config saved to /var/cache/conftool/dbconfig/20220908-113429-root.json
* 11:34 marostegui@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34266 and previous config saved to /var/cache/conftool/dbconfig/20220908-113425-root.json
* 11:34 marostegui@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34265 and previous config saved to /var/cache/conftool/dbconfig/20220908-113417-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2025 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34264 and previous config saved to /var/cache/conftool/dbconfig/20220908-112357-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2022 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34263 and previous config saved to /var/cache/conftool/dbconfig/20220908-112351-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34262 and previous config saved to /var/cache/conftool/dbconfig/20220908-112345-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34261 and previous config saved to /var/cache/conftool/dbconfig/20220908-112341-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2034 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34260 and previous config saved to /var/cache/conftool/dbconfig/20220908-112329-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2033 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34259 and previous config saved to /var/cache/conftool/dbconfig/20220908-112324-root.json
* 11:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2032 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34258 and previous config saved to /var/cache/conftool/dbconfig/20220908-112319-root.json
* 11:19 marostegui@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34257 and previous config saved to /var/cache/conftool/dbconfig/20220908-111924-root.json
* 11:19 marostegui@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34256 and previous config saved to /var/cache/conftool/dbconfig/20220908-111920-root.json
* 11:19 marostegui@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34255 and previous config saved to /var/cache/conftool/dbconfig/20220908-111912-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2025 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34254 and previous config saved to /var/cache/conftool/dbconfig/20220908-110852-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2022 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34253 and previous config saved to /var/cache/conftool/dbconfig/20220908-110846-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34252 and previous config saved to /var/cache/conftool/dbconfig/20220908-110840-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34251 and previous config saved to /var/cache/conftool/dbconfig/20220908-110836-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2034 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34250 and previous config saved to /var/cache/conftool/dbconfig/20220908-110825-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2033 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34249 and previous config saved to /var/cache/conftool/dbconfig/20220908-110819-root.json
* 11:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2032 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34248 and previous config saved to /var/cache/conftool/dbconfig/20220908-110814-root.json
* 11:04 marostegui@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34247 and previous config saved to /var/cache/conftool/dbconfig/20220908-110419-root.json
* 11:04 marostegui@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34246 and previous config saved to /var/cache/conftool/dbconfig/20220908-110415-root.json
* 11:04 marostegui@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34245 and previous config saved to /var/cache/conftool/dbconfig/20220908-110407-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es2025 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34244 and previous config saved to /var/cache/conftool/dbconfig/20220908-105347-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es2022 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34243 and previous config saved to /var/cache/conftool/dbconfig/20220908-105341-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34242 and previous config saved to /var/cache/conftool/dbconfig/20220908-105335-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34241 and previous config saved to /var/cache/conftool/dbconfig/20220908-105331-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es2034 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34240 and previous config saved to /var/cache/conftool/dbconfig/20220908-105320-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es2033 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34239 and previous config saved to /var/cache/conftool/dbconfig/20220908-105314-root.json
* 10:53 marostegui@cumin1001: dbctl commit (dc=all): 'es2032 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34238 and previous config saved to /var/cache/conftool/dbconfig/20220908-105309-root.json
* 10:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1028 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34237 and previous config saved to /var/cache/conftool/dbconfig/20220908-104914-root.json
* 10:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34236 and previous config saved to /var/cache/conftool/dbconfig/20220908-104910-root.json
* 10:49 marostegui@cumin1001: dbctl commit (dc=all): 'es1026 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34235 and previous config saved to /var/cache/conftool/dbconfig/20220908-104902-root.json
* 10:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1027 es1026 es1028 for upgrade', diff saved to https://phabricator.wikimedia.org/P34234 and previous config saved to /var/cache/conftool/dbconfig/20220908-104152-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es2025 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34233 and previous config saved to /var/cache/conftool/dbconfig/20220908-103842-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es2022 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34232 and previous config saved to /var/cache/conftool/dbconfig/20220908-103836-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1025 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34231 and previous config saved to /var/cache/conftool/dbconfig/20220908-103830-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es1022 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34230 and previous config saved to /var/cache/conftool/dbconfig/20220908-103826-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es2034 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34229 and previous config saved to /var/cache/conftool/dbconfig/20220908-103815-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es2033 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34228 and previous config saved to /var/cache/conftool/dbconfig/20220908-103809-root.json
* 10:38 marostegui@cumin1001: dbctl commit (dc=all): 'es2032 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34227 and previous config saved to /var/cache/conftool/dbconfig/20220908-103804-root.json
* 10:31 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 10:30 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 10:29 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 10:29 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34226 and previous config saved to /var/cache/conftool/dbconfig/20220908-102859-root.json
* 10:28 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
* 10:27 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 10:26 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 10:23 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 10:23 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 10:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2034 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34225 and previous config saved to /var/cache/conftool/dbconfig/20220908-102310-root.json
* 10:23 marostegui@cumin1001: dbctl commit (dc=all): 'es2033 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34224 and previous config saved to /var/cache/conftool/dbconfig/20220908-102304-root.json
* 10:22 marostegui@cumin1001: dbctl commit (dc=all): 'es2032 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34223 and previous config saved to /var/cache/conftool/dbconfig/20220908-102259-root.json
* 10:20 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es1022, es1025, es2022, es2025 for upgrade', diff saved to https://phabricator.wikimedia.org/P34222 and previous config saved to /var/cache/conftool/dbconfig/20220908-102040-root.json
* 10:18 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on 15 hosts with reason: Downtiming replaced wtp servers
* 10:18 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on 15 hosts with reason: Downtiming replaced wtp servers
* 10:18 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on 7 hosts with reason: Downtiming replaced wtp servers
* 10:18 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on 7 hosts with reason: Downtiming replaced wtp servers
* 10:13 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34221 and previous config saved to /var/cache/conftool/dbconfig/20220908-101329-root.json
* 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2034 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34220 and previous config saved to /var/cache/conftool/dbconfig/20220908-100805-root.json
* 10:08 marostegui@cumin1001: dbctl commit (dc=all): 'es2033 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34219 and previous config saved to /var/cache/conftool/dbconfig/20220908-100759-root.json
* 10:07 marostegui@cumin1001: dbctl commit (dc=all): 'es2032 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34218 and previous config saved to /var/cache/conftool/dbconfig/20220908-100754-root.json
* 10:07 XioNoX: re-pool esams after routers upgrade - [[phab:T295690|T295690]]
* 10:06 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1029.eqiad.wmnet
* 10:06 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1031.eqiad.wmnet
* 10:06 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1030.eqiad.wmnet
* 10:06 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wtp[1029-1031].eqiad.wmnet with reason: Downtiming replaced wtp servers
* 10:05 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wtp[1029-1031].eqiad.wmnet with reason: Downtiming replaced wtp servers
* 10:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es[2032-2034].codfw.wmnet with reason: Upgrade
* 10:01 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on es[2032-2034].codfw.wmnet with reason: Upgrade
* 10:01 claime: Serving 100% of parsoid traffic with php 7.4 [[phab:T307219|T307219]]
* 10:00 claime: depooled wtp1033.eqiad.wmnet from parsoid cluster [[phab:T307219|T307219]]
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'es2029 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34217 and previous config saved to /var/cache/conftool/dbconfig/20220908-100028-root.json
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'es2030 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34216 and previous config saved to /var/cache/conftool/dbconfig/20220908-100027-root.json
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'es2031 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34215 and previous config saved to /var/cache/conftool/dbconfig/20220908-100025-root.json
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2032 es2033 es2034 for upgrade', diff saved to https://phabricator.wikimedia.org/P34214 and previous config saved to /var/cache/conftool/dbconfig/20220908-100014-root.json
* 09:58 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34213 and previous config saved to /var/cache/conftool/dbconfig/20220908-095759-root.json
* 09:50 claime: pooled parse1024.eqiad.wmnet (php 7.4 only) in parsoid cluster [[phab:T307219|T307219]]
* 09:50 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1024.eqiad.wmnet
* 09:50 cgoubert@cumin1001: START - Cookbook sre.hosts.remove-downtime for parse1024.eqiad.wmnet
* 09:47 claime: depooled wtp1032.eqiad.wmnet from parsoid cluster [[phab:T307219|T307219]]
* 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'es2029 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34212 and previous config saved to /var/cache/conftool/dbconfig/20220908-094523-root.json
* 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'es2030 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34211 and previous config saved to /var/cache/conftool/dbconfig/20220908-094522-root.json
* 09:45 marostegui@cumin1001: dbctl commit (dc=all): 'es2031 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34210 and previous config saved to /var/cache/conftool/dbconfig/20220908-094520-root.json
* 09:42 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34209 and previous config saved to /var/cache/conftool/dbconfig/20220908-094229-root.json
* 09:38 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1024.eqiad.wmnet
* 09:37 claime: pooled parse1023.eqiad.wmnet (php 7.4 only) in parsoid cluster [[phab:T307219|T307219]]
* 09:36 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1023.eqiad.wmnet
* 09:36 cgoubert@cumin1001: START - Cookbook sre.hosts.remove-downtime for parse1023.eqiad.wmnet
* 09:35 XioNoX: drain draffic from cr3-knams - [[phab:T295690|T295690]]
* 09:33 claime: depooled wtp1031.eqiad.wmnet from parsoid cluster [[phab:T307219|T307219]]
* 09:31 vgutierrez: rolling restart of purged - [[phab:T317064|T317064]]
* 09:31 ayounsi@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-knams,cr3-knams IPv6 with reason: router upgrade
* 09:31 ayounsi@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cr3-knams,cr3-knams IPv6 with reason: router upgrade
* 09:31 vgutierrez: upload purged 0.18 to apt.wm.o (buster) - [[phab:T317064|T317064]]
* 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'es2029 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34208 and previous config saved to /var/cache/conftool/dbconfig/20220908-093018-root.json
* 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'es2030 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34207 and previous config saved to /var/cache/conftool/dbconfig/20220908-093017-root.json
* 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'es2031 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34206 and previous config saved to /var/cache/conftool/dbconfig/20220908-093015-root.json
* 09:27 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34205 and previous config saved to /var/cache/conftool/dbconfig/20220908-092700-root.json
* 09:24 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2027 to es3 codfw master', diff saved to https://phabricator.wikimedia.org/P34204 and previous config saved to /var/cache/conftool/dbconfig/20220908-092436-marostegui.json
* 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2026 to es2 codfw master', diff saved to https://phabricator.wikimedia.org/P34203 and previous config saved to /var/cache/conftool/dbconfig/20220908-092346-marostegui.json
* 09:23 marostegui@cumin1001: dbctl commit (dc=all): 'Promote es2028 to es1 codfw master', diff saved to https://phabricator.wikimedia.org/P34202 and previous config saved to /var/cache/conftool/dbconfig/20220908-092301-marostegui.json
* 09:21 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1023.eqiad.wmnet
* 09:21 claime: pooled parse1022.eqiad.wmnet (php 7.4 only) in parsoid cluster [[phab:T307219|T307219]]
* 09:19 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1022.eqiad.wmnet
* 09:19 cgoubert@cumin1001: START - Cookbook sre.hosts.remove-downtime for parse1022.eqiad.wmnet
* 09:18 vgutierrez: testing purged 0.18 in cp4026 and cp4032
* 09:18 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1025.eqiad.wmnet
* 09:17 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1026.eqiad.wmnet
* 09:17 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=wtp1029.eqiad.wmnet
* 09:16 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1029.eqiad.wmnet
* 09:16 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wtp[1025-1028].eqiad.wmnet with reason: Downtiming replaced wtp servers
* 09:16 cgoubert@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=parsoid,name=wtp1028.eqiad.wmnet
* 09:16 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wtp[1025-1028].eqiad.wmnet with reason: Downtiming replaced wtp servers
* 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'es2029 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34201 and previous config saved to /var/cache/conftool/dbconfig/20220908-091513-root.json
* 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'es2030 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34200 and previous config saved to /var/cache/conftool/dbconfig/20220908-091512-root.json
* 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'es2031 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34199 and previous config saved to /var/cache/conftool/dbconfig/20220908-091510-root.json
* 09:14 claime: depooled wtp1030.eqiad.wmnet from parsoid cluster [[phab:T307219|T307219]]
* 09:12 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34198 and previous config saved to /var/cache/conftool/dbconfig/20220908-091200-root.json
* 09:11 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34197 and previous config saved to /var/cache/conftool/dbconfig/20220908-091157-root.json
* 09:11 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34196 and previous config saved to /var/cache/conftool/dbconfig/20220908-091151-root.json
* 09:11 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34195 and previous config saved to /var/cache/conftool/dbconfig/20220908-091129-root.json
* 09:10 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 09:09 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 09:08 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 09:07 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 09:05 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1022.eqiad.wmnet
* 09:04 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 09:04 claime: pooled parse1021.eqiad.wmnet (php 7.4 only) in parsoid cluster [[phab:T307219|T307219]]
* 09:03 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 09:03 cgoubert@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1021.eqiad.wmnet
* 09:03 cgoubert@cumin2002: START - Cookbook sre.hosts.remove-downtime for parse1021.eqiad.wmnet
* 09:02 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'es2029 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34194 and previous config saved to /var/cache/conftool/dbconfig/20220908-090008-root.json
* 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'es2030 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34193 and previous config saved to /var/cache/conftool/dbconfig/20220908-090007-root.json
* 09:00 marostegui@cumin1001: dbctl commit (dc=all): 'es2031 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34192 and previous config saved to /var/cache/conftool/dbconfig/20220908-090005-root.json
* 08:56 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34190 and previous config saved to /var/cache/conftool/dbconfig/20220908-085630-root.json
* 08:56 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34189 and previous config saved to /var/cache/conftool/dbconfig/20220908-085627-root.json
* 08:56 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34188 and previous config saved to /var/cache/conftool/dbconfig/20220908-085621-root.json
* 08:56 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 4%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34187 and previous config saved to /var/cache/conftool/dbconfig/20220908-085559-root.json
* 08:55 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1021.eqiad.wmnet
* 08:53 claime: depooled wtp1029.eqiad.wmnet from parsoid cluster [[phab:T307219|T307219]]
* 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1011.eqiad.wmnet
* 08:47 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-tool1011.eqiad.wmnet
* 08:45 ayounsi@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,re0.cr2-esams.mgmt with reason: router upgrade
* 08:45 marostegui@cumin1001: dbctl commit (dc=all): 'es2029 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34186 and previous config saved to /var/cache/conftool/dbconfig/20220908-084503-root.json
* 08:45 marostegui@cumin1001: dbctl commit (dc=all): 'es2030 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34185 and previous config saved to /var/cache/conftool/dbconfig/20220908-084502-root.json
* 08:45 marostegui@cumin1001: dbctl commit (dc=all): 'es2031 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34184 and previous config saved to /var/cache/conftool/dbconfig/20220908-084500-root.json
* 08:44 ayounsi@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cr2-esams,cr2-esams IPv6,re0.cr2-esams.mgmt with reason: router upgrade
* 08:44 XioNoX: reverting cr3-esams changes (JTAC will be needed for a firmware upgrade), and moving on to cr2-esams - [[phab:T295690|T295690]]
* 08:41 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 100%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34183 and previous config saved to /var/cache/conftool/dbconfig/20220908-084133-root.json
* 08:41 claime: pooled parse1020.eqiad.wmnet (php 7.4 only) in parsoid cluster [[phab:T307219|T307219]]
* 08:41 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34182 and previous config saved to /var/cache/conftool/dbconfig/20220908-084059-root.json
* 08:40 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34181 and previous config saved to /var/cache/conftool/dbconfig/20220908-084057-root.json
* 08:40 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34180 and previous config saved to /var/cache/conftool/dbconfig/20220908-084051-root.json
* 08:40 cgoubert@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1020.eqiad.wmnet
* 08:40 cgoubert@cumin2002: START - Cookbook sre.hosts.remove-downtime for parse1020.eqiad.wmnet
* 08:40 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 3%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34179 and previous config saved to /var/cache/conftool/dbconfig/20220908-084029-root.json
* 08:40 claime: depooled wtp1028.eqiad.wmnet from parsoid cluster [[phab:T307219|T307219]]
* 08:39 marostegui@cumin2002: dbctl commit (dc=all): 'Depool es2029, es2030, es2031', diff saved to https://phabricator.wikimedia.org/P34178 and previous config saved to /var/cache/conftool/dbconfig/20220908-083941-marostegui.json
* 08:31 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1020.eqiad.wmnet
* 08:26 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 75%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34176 and previous config saved to /var/cache/conftool/dbconfig/20220908-082604-root.json
* 08:25 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34175 and previous config saved to /var/cache/conftool/dbconfig/20220908-082530-root.json
* 08:25 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34174 and previous config saved to /var/cache/conftool/dbconfig/20220908-082528-root.json
* 08:25 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34173 and previous config saved to /var/cache/conftool/dbconfig/20220908-082521-root.json
* 08:25 marostegui@cumin2002: dbctl commit (dc=all): 'db1127 (re)pooling @ 2%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34172 and previous config saved to /var/cache/conftool/dbconfig/20220908-082500-root.json
* 08:24 claime: pooled parse1019.eqiad.wmnet (php 7.4 only) in parsoid cluster [[phab:T307219|T307219]]
* 08:22 cgoubert@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for parse1019.eqiad.wmnet
* 08:22 cgoubert@cumin2002: START - Cookbook sre.hosts.remove-downtime for parse1019.eqiad.wmnet
* 08:12 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin1001.eqiad.wmnet
* 08:10 ayounsi@cumin2002: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 08:10 ayounsi@cumin2002: START - Cookbook sre.network.cf
* 08:10 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 50%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34171 and previous config saved to /var/cache/conftool/dbconfig/20220908-081034-root.json
* 08:09 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34170 and previous config saved to /var/cache/conftool/dbconfig/20220908-080958-root.json
* 08:09 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34169 and previous config saved to /var/cache/conftool/dbconfig/20220908-080951-root.json
* 08:09 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34168 and previous config saved to /var/cache/conftool/dbconfig/20220908-080946-root.json
* 08:09 ayounsi@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-esams,cr3-esams IPv6,re0.cr3-esams.mgmt with reason: router upgrade
* 08:09 ayounsi@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cr3-esams,cr3-esams IPv6,re0.cr3-esams.mgmt with reason: router upgrade
* 08:08 cgoubert@puppetmaster1001: conftool action : set/pooled=no:weight=10; selector: dc=eqiad,cluster=parsoid,name=parse1019.eqiad.wmnet
* 08:08 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 100%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34167 and previous config saved to /var/cache/conftool/dbconfig/20220908-080823-root.json
* 08:07 XioNoX: drain draffic from cr3-esams - [[phab:T295690|T295690]]
* 08:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host cumin1001.eqiad.wmnet
* 07:55 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 25%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34166 and previous config saved to /var/cache/conftool/dbconfig/20220908-075504-root.json
* 07:54 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34165 and previous config saved to /var/cache/conftool/dbconfig/20220908-075429-root.json
* 07:54 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34164 and previous config saved to /var/cache/conftool/dbconfig/20220908-075421-root.json
* 07:54 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34163 and previous config saved to /var/cache/conftool/dbconfig/20220908-075416-root.json
* 07:52 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 75%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34162 and previous config saved to /var/cache/conftool/dbconfig/20220908-075253-root.json
* 07:41 XioNoX: depool esams for routers upgrade - [[phab:T295690|T295690]]
* 07:39 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 10%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34161 and previous config saved to /var/cache/conftool/dbconfig/20220908-073935-root.json
* 07:39 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 4%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34160 and previous config saved to /var/cache/conftool/dbconfig/20220908-073900-root.json
* 07:38 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 4%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34159 and previous config saved to /var/cache/conftool/dbconfig/20220908-073851-root.json
* 07:38 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 4%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34158 and previous config saved to /var/cache/conftool/dbconfig/20220908-073846-root.json
* 07:37 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 50%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34157 and previous config saved to /var/cache/conftool/dbconfig/20220908-073724-root.json
* 07:24 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 5%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34156 and previous config saved to /var/cache/conftool/dbconfig/20220908-072405-root.json
* 07:23 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 3%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34155 and previous config saved to /var/cache/conftool/dbconfig/20220908-072330-root.json
* 07:23 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 3%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34154 and previous config saved to /var/cache/conftool/dbconfig/20220908-072321-root.json
* 07:23 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 3%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34153 and previous config saved to /var/cache/conftool/dbconfig/20220908-072316-root.json
* 07:21 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 25%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34152 and previous config saved to /var/cache/conftool/dbconfig/20220908-072154-root.json
* 07:08 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 4%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34151 and previous config saved to /var/cache/conftool/dbconfig/20220908-070836-root.json
* 07:08 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 2%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34150 and previous config saved to /var/cache/conftool/dbconfig/20220908-070800-root.json
* 07:07 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 2%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34149 and previous config saved to /var/cache/conftool/dbconfig/20220908-070752-root.json
* 07:07 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 2%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34148 and previous config saved to /var/cache/conftool/dbconfig/20220908-070746-root.json
* 07:06 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 10%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34147 and previous config saved to /var/cache/conftool/dbconfig/20220908-070625-root.json
* 07:01 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 07:01 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 07:01 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 07:00 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 07:00 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 07:00 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 06:53 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 3%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34146 and previous config saved to /var/cache/conftool/dbconfig/20220908-065306-root.json
* 06:52 marostegui@cumin2002: dbctl commit (dc=all): 'es2028 (re)pooling @ 1%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34145 and previous config saved to /var/cache/conftool/dbconfig/20220908-065229-root.json
* 06:52 marostegui@cumin2002: dbctl commit (dc=all): 'es2027 (re)pooling @ 1%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34144 and previous config saved to /var/cache/conftool/dbconfig/20220908-065222-root.json
* 06:52 marostegui@cumin2002: dbctl commit (dc=all): 'es2026 (re)pooling @ 1%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34143 and previous config saved to /var/cache/conftool/dbconfig/20220908-065216-root.json
* 06:50 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 5%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34142 and previous config saved to /var/cache/conftool/dbconfig/20220908-065054-root.json
* 06:44 marostegui@cumin2002: dbctl commit (dc=all): 'Depool es2026, es2027, es2028', diff saved to https://phabricator.wikimedia.org/P34141 and previous config saved to /var/cache/conftool/dbconfig/20220908-064450-marostegui.json
* 06:37 marostegui@cumin2002: dbctl commit (dc=all): 'db1203 (re)pooling @ 2%: Pooling for the first time in s8', diff saved to https://phabricator.wikimedia.org/P34140 and previous config saved to /var/cache/conftool/dbconfig/20220908-063737-root.json
* 06:35 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 4%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34139 and previous config saved to /var/cache/conftool/dbconfig/20220908-063525-root.json
* 06:19 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 3%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34138 and previous config saved to /var/cache/conftool/dbconfig/20220908-061955-root.json
* 06:14 marostegui@cumin2002: dbctl commit (dc=all): 'Add db1203 to s8, depooled, [[phab:T316342|T316342]]', diff saved to https://phabricator.wikimedia.org/P34137 and previous config saved to /var/cache/conftool/dbconfig/20220908-061413-marostegui.json
* 06:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 06:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 06:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depool db1157 [[phab:T316622|T316622]]', diff saved to https://phabricator.wikimedia.org/P34136 and previous config saved to /var/cache/conftool/dbconfig/20220908-060438-ladsgroup.json
* 06:04 marostegui@cumin2002: dbctl commit (dc=all): 'db1202 (re)pooling @ 2%: Pooling for the first time in s7', diff saved to https://phabricator.wikimedia.org/P34135 and previous config saved to /var/cache/conftool/dbconfig/20220908-060426-root.json
* 06:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Promote db1123 to s3 primary and set section read-write [[phab:T316622|T316622]]', diff saved to https://phabricator.wikimedia.org/P34134 and previous config saved to /var/cache/conftool/dbconfig/20220908-060138-ladsgroup.json
* 06:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - [[phab:T316622|T316622]]', diff saved to https://phabricator.wikimedia.org/P34133 and previous config saved to /var/cache/conftool/dbconfig/20220908-060110-ladsgroup.json
* 06:00 Amir1: Starting s3 eqiad failover from db1157 to db1123 - [[phab:T316622|T316622]]
* 05:55 marostegui@cumin1001: dbctl commit (dc=all): 'Increase weight for db1194', diff saved to https://phabricator.wikimedia.org/P34132 and previous config saved to /var/cache/conftool/dbconfig/20220908-055546-marostegui.json
* 05:54 marostegui@cumin1001: dbctl commit (dc=all): 'Pooling db1202 for the first time in s7 [[phab:T316342|T316342]]', diff saved to https://phabricator.wikimedia.org/P34131 and previous config saved to /var/cache/conftool/dbconfig/20220908-055451-marostegui.json
* 05:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Pooling back db2140', diff saved to https://phabricator.wikimedia.org/P34130 and previous config saved to /var/cache/conftool/dbconfig/20220908-054921-ladsgroup.json
* 05:44 marostegui@cumin1001: dbctl commit (dc=all): 'Add db1202 to s7, depooled, [[phab:T316342|T316342]]', diff saved to https://phabricator.wikimedia.org/P34129 and previous config saved to /var/cache/conftool/dbconfig/20220908-054429-marostegui.json
* 05:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Set db1123 with weight 0 [[phab:T316622|T316622]]', diff saved to https://phabricator.wikimedia.org/P34128 and previous config saved to /var/cache/conftool/dbconfig/20220908-051043-ladsgroup.json
* 05:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s3 [[phab:T316622|T316622]]
* 05:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1:00:00 on 23 hosts with reason: Primary switchover s3 [[phab:T316622|T316622]]
* 02:04 pt1979@cumin1001: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging1005:
* 02:04 pt1979@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging1005:
* 02:02 pt1979@cumin1001: END (ERROR) - Cookbook sre.network.configure-switch-interfaces (exit_code=97) for host db2169
* 02:02 pt1979@cumin1001: START - Cookbook sre.network.configure-switch-interfaces for host db2169
* 02:01 pt1979@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 01:59 pt1979@cumin1001: START - Cookbook sre.dns.netbox
* 01:56 ejegg: re-enabled recurring charge job
* 01:48 ejegg: updated fundraising civicrm from {{Gerrit|c1f0e041}} to {{Gerrit|efbbcb57}}
* 01:26 ejegg: disabled recurring charge job
* 00:33 ejegg: updated standalone Smashpig from {{Gerrit|11ba0a1b}} to {{Gerrit|88e5e9bb}}
== 2022-09-07 ==
* 22:12 bd808: Attempting to migrate all remaining Striker managed git repos from Diffusion to GitLab ([[phab:T315706|T315706]])
* 21:27 TheresNoTime: closing UTC late backport window, +27m
* 21:26 samtar@deploy1002: Finished scap: Backport for [[gerrit:830602{{!}}Respect skin's TOC option (T316947)]] (duration: 08m 02s)
* 21:25 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:24 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:24 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:19 samtar@deploy1002: samtar and jdlrobson: Backport for [[gerrit:830602{{!}}Respect skin's TOC option (T316947)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 21:18 samtar@deploy1002: Started scap: Backport for [[gerrit:830602{{!}}Respect skin's TOC option (T316947)]]
* 21:14 samtar@deploy1002: Finished scap: Backport for [[gerrit:830601{{!}}Respect skin's TOC option (T316947)]] (duration: 07m 06s)
* 21:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:07 samtar@deploy1002: samtar and jdlrobson: Backport for [[gerrit:830601{{!}}Respect skin's TOC option (T316947)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 21:07 samtar@deploy1002: Started scap: Backport for [[gerrit:830601{{!}}Respect skin's TOC option (T316947)]]
* 21:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:02 samtar@deploy1002: Finished scap: Backport for [[gerrit:830641{{!}}beta: Remove deployment-parsoid11 from wgLinterSubmitterWhitelist]] (duration: 04m 36s)
* 21:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:02 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:01 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 21:01 TheresNoTime: extending UTC late backport window
* 21:01 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
* 20:58 samtar@deploy1002: samtar and zabe: Backport for [[gerrit:830641{{!}}beta: Remove deployment-parsoid11 from wgLinterSubmitterWhitelist]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 20:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 20:58 samtar@deploy1002: Started scap: Backport for [[gerrit:830641{{!}}beta: Remove deployment-parsoid11 from wgLinterSubmitterWhitelist]]
* 20:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
* 20:56 samtar@deploy1002: Finished scap: Backport for [[gerrit:830313{{!}}Enable Extension:Nearby on wikidata (T246493)]] (duration: 05m 54s)
* 20:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25816 and previous config saved to /var/cache/conftool/dbconfig/20220420-205732-ladsgroup.json
* 20:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:46 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudcephmon2006-dev.codfw.wmnet with OS buster
* 20:55 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:46 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS buster
* 20:55 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:46 jhuneidi@deploy1002: Synchronized static/images/project-logos/: Config: [[gerrit:784710{{!}}Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)"]] (duration: 00m 51s)
* 20:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:44 jhuneidi@deploy1002: Synchronized static/images/mobile/copyright/: Config: [[gerrit:784708{{!}}Revert "fawiki: Change logo for 900K milestone"]] (duration: 00m 49s)
* 20:51 samtar@deploy1002: samtar and jdlrobson: Backport for [[gerrit:830313{{!}}Enable Extension:Nearby on wikidata (T246493)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 20:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25815 and previous config saved to /var/cache/conftool/dbconfig/20220420-204227-ladsgroup.json
* 20:50 samtar@deploy1002: Started scap: Backport for [[gerrit:830313{{!}}Enable Extension:Nearby on wikidata (T246493)]]
* 20:40 jhuneidi@deploy1002: Synchronized wmf-config/logos.php: Config: [[gerrit:784710{{!}}Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)"]] (duration: 00m 50s)
* 20:49 samtar@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:830312{{!}}Wikidata has a wordmark (T315572)]] (duration: 03m 44s)
* 20:38 jhuneidi@deploy1002: Synchronized logos/config.yaml: Config: [[gerrit:784710{{!}}Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)"]] (duration: 00m 51s)
* 20:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:37 jhuneidi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784708{{!}}Revert "fawiki: Change logo for 900K milestone"]] [[gerrit:784710{{!}}Revert "fawiki: Change wordmark & tagline (new Vector) and logo (legacy Vector)"]] (duration: 00m 57s)
* 20:44 samtar@deploy1002: Synchronized static/images/mobile/copyright/wikidata-en.svg: Config: [[gerrit:830312{{!}}Wikidata has a wordmark (T315572)]] (duration: 03m 45s)
* 20:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:37 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:36 mutante: gitlab-runner2001 - mkdir /home/gitlab-runner  (was: PANIC: mkdir /home/gitlab-runner: permission denied and other issues, trying if it's just the missing directory or more) [[phab:T297659|T297659]]
* 20:35 samtar@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:830600{{!}}Revert "Enable wgDiscussionToolsEnablePermalinksBackend on all wikis"]] (duration: 03m 42s)
* 20:27 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25814 and previous config saved to /var/cache/conftool/dbconfig/20220420-202722-ladsgroup.json
* 20:21 samtar@deploy1002: Finished scap: Backport for [[gerrit:830687{{!}}Enable wgDiscussionToolsEnablePermalinksBackend on all wikis (T315353)]] (duration: 06m 57s)
* 20:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25813 and previous config saved to /var/cache/conftool/dbconfig/20220420-201240-ladsgroup.json
* 20:14 samtar@deploy1002: samtar and matmarex: Backport for [[gerrit:830687{{!}}Enable wgDiscussionToolsEnablePermalinksBackend on all wikis (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 20:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 20:14 samtar@deploy1002: Started scap: Backport for [[gerrit:830687{{!}}Enable wgDiscussionToolsEnablePermalinksBackend on all wikis (T315353)]]
* 20:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 20:12 mutante: pcc-worker1003 - rm of /srv/jenkins/puppet-compiler/output/36713 and 37153 - /srv is back to 58% usage again
* 20:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25812 and previous config saved to /var/cache/conftool/dbconfig/20220420-201232-ladsgroup.json
* 20:10 mutante: integration.wikimedia.org - clicked to delete builds 36713 and 37153 because they were several GB in size
* 20:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25811 and previous config saved to /var/cache/conftool/dbconfig/20220420-201217-ladsgroup.json
* 20:08 TheresNoTime: running `extensions/WikimediaMaintenance/createExtensionTables.php discussiontools` on mwmaint1002
* 20:10 gmodena@deploy1002: Finished deploy [airflow-dags/research@b029f10]: (no justification provided) (duration: 00m 06s)
* 20:08 mutante: puppet compiler out of disk space, (pcc-worker1003): identified build 37153 as huge compared to others in the filesystem, then clicked to delete it via integration.wm.org web UI
* 20:10 gmodena@deploy1002: Started deploy [airflow-dags/research@b029f10]: (no justification provided)
* 19:26 herron@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons.
* 19:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25810 and previous config saved to /var/cache/conftool/dbconfig/20220420-195727-ladsgroup.json
* 18:34 dduvall@deploy1002: Finished deploy [phabricator/deployment@a7616e6]: testing deployment to phab2001 (inactive) (duration: 00m 35s)
* 19:56
* 18:33 dduvall@deploy1002: Started deploy [phabricator/deployment@a7616e6]: testing deployment to phab2001 (inactive)
* 18:26 herron@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons.
* 18:25 herron@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-codfw cluster: Roll restart of jvm daemons.
* 17:24 herron@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-logging-codfw cluster: Roll restart of jvm daemons.
* 16:56 ejegg: restarted fundraising scheduled jobs
* 16:34 ejegg: fundraising civicrm upgraded from {{Gerrit|2fcd3bb4}} to {{Gerrit|c1f0e041}}
* 16:32 xcollazo@deploy1002: Finished deploy [airflow-dags/platform_eng@9e4ed94]: Update platform_eng Airflow to latest (duration: 00m 10s)
* 16:31 xcollazo@deploy1002: Started deploy [airflow-dags/platform_eng@9e4ed94]: Update platform_eng Airflow to latest
* 16:31 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 16:30 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 16:22 moritzm: installing twisted security updates on bullseye
* 16:22 cparle@deploy1002: Finished deploy [airflow-dags/platform_eng@9e4ed94]: (no justification provided) (duration: 00m 17s)
* 16:21 cparle@deploy1002: Started deploy [airflow-dags/platform_eng@9e4ed94]: (no justification provided)
* 16:16 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 16:00 ejegg: fundraising civicrm upgraded from {{Gerrit|5aa1309d}} to {{Gerrit|2fcd3bb4}}
* 15:55 ejegg: fundraising scheduled jobs disabled for deployment
* 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'db1173 (re)
* 07:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 07:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
* 06:28 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox interface ID cr2-eqiad:xe-4/1/3
* 07:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance
* 06:28 ayounsi@cumin1001: START - Cookbook sre.network.debug for Netbox interface ID cr2-eqiad:xe-4/1/3
* 07:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25560 and previous config saved to /var/cache/conftool/dbconfig/20220420-070721-marostegui.json
* 06:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 07:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25559 and previous config saved to /var/cache/conftool/dbconfig/20220420-070702-ladsgroup.json
* 06:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1119.eqiad.wmnet with reason: Maintenance
* 07:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 02:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 02:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2099.codfw.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 02:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P33778 and previous config saved to /var/cache/conftool/dbconfig/20220905-024602-ladsgroup.json
* 07:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 00:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25558 and previous config saved to /var/cache/conftool/dbconfig/20220420-070648-ladsgroup.json
* 00:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1107.eqiad.wmnet with reason: Maintenance
* 07:05 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 00:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33777 and previous config saved to /var/cache/conftool/dbconfig/20220905-003619-ladsgroup.json
* 07:02 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P33776 and previous config saved to /var/cache/conftool/dbconfig/20220905-002112-ladsgroup.json
* 07:00 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 00:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P33775 and previous config saved to /var/cache/conftool/dbconfig/20220905-000606-ladsgroup.json
* 06:59 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 06:57 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 06:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P25557 and previous config saved to /var/cache/conftool/dbconfig/20220420-065216-marostegui.json
* 06:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25556 and previous config saved to /var/cache/conftool/dbconfig/20220420-065143-ladsgroup.json
* 06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P25555 and previous config saved to /var/cache/conftool/dbconfig/20220420-063711-marostegui.json
* 06:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25554 and previous config saved to /var/cache/conftool/dbconfig/20220420-063638-ladsgroup.json
* 06:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 06:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 06:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 06:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 06:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25553 and previous config saved to /var/cache/conftool/dbconfig/20220420-062206-marostegui.json
* 06:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25552 and previous config saved to /var/cache/conftool/dbconfig/20220420-062133-ladsgroup.json
* 06:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25551 and previous config saved to /var/cache/conftool/dbconfig/20220420-061848-marostegui.json
* 06:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 06:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 06:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 06:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 06:18 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 06:18 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25550 and previous config saved to /var/cache/conftool/dbconfig/20220420-061834-marostegui.json
* 06:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25549 and previous config saved to /var/cache/conftool/dbconfig/20220420-061433-ladsgroup.json
* 06:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 06:14 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 06:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25548 and previous config saved to /var/cache/conftool/dbconfig/20220420-061425-ladsgroup.json
* 06:07 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P25547 and previous config saved to /var/cache/conftool/dbconfig/20220420-060732-root.json
* 06:03 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P25546 and previous config saved to /var/cache/conftool/dbconfig/20220420-060329-marostegui.json
* 05:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25545 and previous config saved to /var/cache/conftool/dbconfig/20220420-055920-ladsgroup.json
* 05:52 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P25544 and previous config saved to /var/cache/conftool/dbconfig/20220420-055228-root.json
* 05:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P25543 and previous config saved to /var/cache/conftool/dbconfig/20220420-054824-marostegui.json
* 05:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25542 and previous config saved to /var/cache/conftool/dbconfig/20220420-054415-ladsgroup.json
* 05:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 05:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 05:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25541 and previous config saved to /var/cache/conftool/dbconfig/20220420-053932-ladsgroup.json
* 05:39 ayounsi@cumin2002: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 05:39 ayounsi@cumin2002: START - Cookbook sre.network.cf
* 05:38 XioNoX: start CF in monitoring mode for drmrs
* 05:37 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P25540 and previous config saved to /var/cache/conftool/dbconfig/20220420-053724-root.json
* 05:33 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1163 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25539 and previous config saved to /var/cache/conftool/dbconfig/20220420-053319-marostegui.json
* 05:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1163 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25538 and previous config saved to /var/cache/conftool/dbconfig/20220420-053006-marostegui.json
* 05:30 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 05:30 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 05:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25537 and previous config saved to /var/cache/conftool/dbconfig/20220420-052958-marostegui.json
* 05:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25536 and previous config saved to /var/cache/conftool/dbconfig/20220420-052910-ladsgroup.json
* 05:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25535 and previous config saved to /var/cache/conftool/dbconfig/20220420-052427-ladsgroup.json
* 05:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25534 and previous config saved to /var/cache/conftool/dbconfig/20220420-052223-ladsgroup.json
* 05:22 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P25533 and previous config saved to /var/cache/conftool/dbconfig/20220420-052220-root.json
* 05:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 05:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 05:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25532 and previous config saved to /var/cache/conftool/dbconfig/20220420-052215-ladsgroup.json
* 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P25531 and previous config saved to /var/cache/conftool/dbconfig/20220420-051453-marostegui.json
* 05:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25530 and previous config saved to /var/cache/conftool/dbconfig/20220420-050921-ladsgroup.json
* 05:07 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P25529 and previous config saved to /var/cache/conftool/dbconfig/20220420-050716-root.json
* 05:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25528 and previous config saved to /var/cache/conftool/dbconfig/20220420-050710-ladsgroup.json
* 04:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P25527 and previous config saved to /var/cache/conftool/dbconfig/20220420-045948-marostegui.json
* 04:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25526 and previous config saved to /var/cache/conftool/dbconfig/20220420-045416-ladsgroup.json
* 04:52 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 5%: After reimage', diff saved to https://phabricator.wikimedia.org/P25525 and previous config saved to /var/cache/conftool/dbconfig/20220420-045212-root.json
* 04:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25524 and previous config saved to /var/cache/conftool/dbconfig/20220420-045205-ladsgroup.json
* 04:51 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1132 into s1 [[phab:T301879|T301879]]', diff saved to https://phabricator.wikimedia.org/P25523 and previous config saved to /var/cache/conftool/dbconfig/20220420-045108-marostegui.json
* 04:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1169 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25522 and previous config saved to /var/cache/conftool/dbconfig/20220420-044443-marostegui.json
* 04:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1169 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25521 and previous config saved to /var/cache/conftool/dbconfig/20220420-044132-marostegui.json
* 04:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
* 04:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
* 04:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 04:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
* 04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25520 and previous config saved to /var/cache/conftool/dbconfig/20220420-043711-ladsgroup.json
* 04:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 04:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 04:37 marostegui@cumin1001: dbctl commit (dc=all): 'db1136 (re)pooling @ 1%: After reimage', diff saved to https://phabricator.wikimedia.org/P25519 and previous config saved to /var/cache/conftool/dbconfig/20220420-043702-root.json
* 04:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25518 and previous config saved to /var/cache/conftool/dbconfig/20220420-043700-ladsgroup.json
* 04:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25517 and previous config saved to /var/cache/conftool/dbconfig/20220420-043005-ladsgroup.json
* 04:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 04:29 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 04:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25516 and previous config saved to /var/cache/conftool/dbconfig/20220420-042152-ladsgroup.json
* 04:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25515 and previous config saved to /var/cache/conftool/dbconfig/20220420-040647-ladsgroup.json
* 03:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25514 and previous config saved to /var/cache/conftool/dbconfig/20220420-035142-ladsgroup.json
* 03:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25513 and previous config saved to /var/cache/conftool/dbconfig/20220420-034443-ladsgroup.json
* 03:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25512 and previous config saved to /var/cache/conftool/dbconfig/20220420-034211-ladsgroup.json
* 03:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 03:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 03:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 03:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 03:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 03:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 03:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 03:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 03:31 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 03:31 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 03:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25511 and previous config saved to /var/cache/conftool/dbconfig/20220420-033126-ladsgroup.json
* 03:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25510 and previous config saved to /var/cache/conftool/dbconfig/20220420-032157-ladsgroup.json
* 03:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 03:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25509 and previous config saved to /var/cache/conftool/dbconfig/20220420-031621-ladsgroup.json
* 03:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25508 and previous config saved to /var/cache/conftool/dbconfig/20220420-030454-ladsgroup.json
* 03:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25507 and previous config saved to /var/cache/conftool/dbconfig/20220420-030116-ladsgroup.json
* 02:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25506 and previous config saved to /var/cache/conftool/dbconfig/20220420-024949-ladsgroup.json
* 02:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25505 and previous config saved to /var/cache/conftool/dbconfig/20220420-024611-ladsgroup.json
* 02:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25504 and previous config saved to /var/cache/conftool/dbconfig/20220420-023951-ladsgroup.json
* 02:39 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 02:39 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 02:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25503 and previous config saved to /var/cache/conftool/dbconfig/20220420-023857-ladsgroup.json
* 02:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25502 and previous config saved to /var/cache/conftool/dbconfig/20220420-023444-ladsgroup.json
* 02:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25501 and previous config saved to /var/cache/conftool/dbconfig/20220420-022352-ladsgroup.json
* 02:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25500 and previous config saved to /var/cache/conftool/dbconfig/20220420-021939-ladsgroup.json
* 02:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25499 and previous config saved to /var/cache/conftool/dbconfig/20220420-020846-ladsgroup.json
* 01:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25498 and previous config saved to /var/cache/conftool/dbconfig/20220420-015341-ladsgroup.json
* 01:31 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 01:28 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25497 and previous config saved to /var/cache/conftool/dbconfig/20220420-011925-ladsgroup.json
* 01:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 01:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 01:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25496 and previous config saved to /var/cache/conftool/dbconfig/20220420-011917-ladsgroup.json
* 01:16 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudweb2002-dev.wikimedia.org with OS buster
* 01:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudweb2002-dev.wikimedia.org with reason: host reimage
* 01:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25495 and previous config saved to /var/cache/conftool/dbconfig/20220420-010412-ladsgroup.json
* 01:01 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudweb2002-dev.wikimedia.org with reason: host reimage
* 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25494 and previous config saved to /var/cache/conftool/dbconfig/20220420-005327-ladsgroup.json
* 00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 00:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 00:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 00:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25493 and previous config saved to /var/cache/conftool/dbconfig/20220420-005314-ladsgroup.json
* 00:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P25492 and previous config saved to /var/cache/conftool/dbconfig/20220420-004907-ladsgroup.json
* 00:46 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS buster
* 00:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudweb2002-dev.wikimedia.org with OS buster
* 00:44 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS buster
* 00:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudservices2005-dev.wikimedia.org with OS bullseye
* 00:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25491 and previous config saved to /var/cache/conftool/dbconfig/20220420-003809-ladsgroup.json
* 00:35 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudweb2002-dev.wikimedia.org with OS buster
* 00:35 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS buster
* 00:34 pt1979@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudweb2002-dev.wikimedia.org with OS bullseye
* 00:34 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudweb2002-dev.wikimedia.org with OS bullseye
* 00:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25490 and previous config saved to /var/cache/conftool/dbconfig/20220420-003401-ladsgroup.json
* 00:28 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices2005-dev.wikimedia.org with reason: host reimage
* 00:25 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices2005-dev.wikimedia.org with reason: host reimage
* 00:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25489 and previous config saved to /var/cache/conftool/dbconfig/20220420-002303-ladsgroup.json
* 00:10 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2006-dev.codfw.wmnet with OS bullseye
* 00:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25488 and previous config saved to /var/cache/conftool/dbconfig/20220420-000758-ladsgroup.json
* 00:06 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudservices2005-dev.wikimedia.org with OS bullseye
* 00:04 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudservices2004-dev.wikimedia.org with OS bullseye
* 00:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25487 and previous config saved to /var/cache/conftool/dbconfig/20220420-000141-ladsgroup.json
* 00:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 00:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 00:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage


== 2022-04-19 ==
== 2022-09-04 ==
* 23:56 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2006-dev.codfw.wmnet with reason: host reimage
* 23:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33774 and previous config saved to /var/cache/conftool/dbconfig/20220904-235100-ladsgroup.json
* 23:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33773 and previous config saved to /var/cache/conftool/dbconfig/20220904-225044-ladsgroup.json
* 23:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 23:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 22:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 23:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 22:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
* 23:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 22:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33772 and previous config saved to /var/cache/conftool/dbconfig/20220904-225016-ladsgroup.json
* 23:54 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices2004-dev.wikimedia.org with reason: host reimage
* 22:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P33771 and previous config saved to /var/cache/conftool/dbconfig/20220904-223510-ladsgroup.json
* 23:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 22:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P33770 and previous config saved to /var/cache/conftool/dbconfig/20220904-222004-ladsgroup.json
* 23:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 22:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33769 and previous config saved to /var/cache/conftool/dbconfig/20220904-220457-ladsgroup.json
* 23:49 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices2004-dev.wikimedia.org with reason: host reimage
* 15:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33767 and previous config saved to /var/cache/conftool/dbconfig/20220904-155059-ladsgroup.json
* 23:34 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudnet2006-dev.codfw.wmnet with OS bullseye
* 15:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 23:34 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet2005-dev.codfw.wmnet with OS bullseye
* 15:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1105.eqiad.wmnet with reason: Maintenance
* 23:30 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudservices2004-dev.wikimedia.org with OS bullseye
* 15:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33766 and previous config saved to /var/cache/conftool/dbconfig/20220904-155027-ladsgroup.json
* 23:28 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet with OS bullseye
* 15:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P33765 and previous config saved to /var/cache/conftool/dbconfig/20220904-153521-ladsgroup.json
* 23:24 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
* 15:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P33764 and previous config saved to /var/cache/conftool/dbconfig/20220904-152015-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25486 and previous config saved to /var/cache/conftool/dbconfig/20220419-232250-ladsgroup.json
* 15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33763 and previous config saved to /var/cache/conftool/dbconfig/20220904-150508-ladsgroup.json
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 12:51 elukey: reset-fail ifup@ens13.service on idp2002
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 12:50 elukey: reset-fail ifup@ens13.service on netflow4002
* 23:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 12:49 elukey: pkill remaining processes of user effeietsanders on stat1008 to unblock puppet - [[phab:T314846|T314846]]
* 23:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance
* 10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db2138:3314 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P33762 and previous config saved to /var/cache/conftool/dbconfig/20220904-103427-ladsgroup.json
* 23:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25485 and previous config saved to /var/cache/conftool/dbconfig/20220419-232237-ladsgroup.json
* 10:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
* 23:20 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2005-dev.codfw.wmnet with reason: host reimage
* 10:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance
* 23:18 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
* 10:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db2136 ([[phab:T314041|T314041]])', diff saved to https://phabricator.wikimedia.org/P33761 and previous config saved to /var/cache/conftool/dbconfig/20220904-103405-ladsgroup.json
* 23:15 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2006-dev.codfw.wmnet with reason: host reimage
* 08:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 ([[phab:T312863|T312863]])', diff saved to https://phabricator.wikimedia.org/P33760 and previous config saved to /var/cache/conftool/dbconfig/20220904-083341-ladsgroup.json
* 23:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25484 and previous config saved to /var/cache/conftool/dbconfig/20220419-230732-ladsgroup.json
* 08:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 23:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25483 and previous config saved to /var/cache/conftool/dbconfig/20220419-230459-ladsgroup.json
* 08:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1099.eqiad.wmnet with reason: Maintenance
* 23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25482 and previous config saved to /var/cache/conftool/dbconfig/20220419-230226-ladsgroup.json
* 23:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 23:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 23:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25481 and previous config saved to /var/cache/conftool/dbconfig/20220419-230218-ladsgroup.json
* 22:56 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudnet2005-dev.codfw.wmnet with OS bullseye
* 22:53 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephmon2006-dev.codfw.wmnet with OS bullseye
* 22:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:53 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:53 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25480 and previous config saved to /var/cache/conftool/dbconfig/20220419-225227-ladsgroup.json
* 22:50 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 22:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25479 and previous config saved to /var/cache/conftool/dbconfig/20220419-224711-ladsgroup.json
* 22:42 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - [[phab:T301959|T301959]]
* 22:40 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 22:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25478 and previous config saved to /var/cache/conftool/dbconfig/20220419-223722-ladsgroup.json
* 22:36 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2005-dev.codfw.wmnet with reason: host reimage
* 22:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25477 and previous config saved to /var/cache/conftool/dbconfig/20220419-223356-ladsgroup.json
* 22:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25476 and previous config saved to /var/cache/conftool/dbconfig/20220419-223206-ladsgroup.json
* 22:22 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 22:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 22:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 22:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 22:18 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.8  refs [[phab:T305214|T305214]]
* 22:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25475 and previous config saved to /var/cache/conftool/dbconfig/20220419-221851-ladsgroup.json
* 22:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25474 and previous config saved to /var/cache/conftool/dbconfig/20220419-221701-ladsgroup.json
* 22:14 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcephmon2005-dev.codfw.wmnet with OS bullseye
* 22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25473 and previous config saved to /var/cache/conftool/dbconfig/20220419-221038-ladsgroup.json
* 22:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 22:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25472 and previous config saved to /var/cache/conftool/dbconfig/20220419-221030-ladsgroup.json
* 22:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25471 and previous config saved to /var/cache/conftool/dbconfig/20220419-220346-ladsgroup.json
* 21:58 ebernhardson: set indices.recovery.max_bytes_per_sec=240mb in elasticsearch-eqiad-psi
* 21:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25470 and previous config saved to /var/cache/conftool/dbconfig/20220419-215525-ladsgroup.json
* 21:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25469 and previous config saved to /var/cache/conftool/dbconfig/20220419-214841-ladsgroup.json
* 21:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 21:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 21:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 21:41 jhuneidi@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/LdapAuthentication/includes/LdapAuthenticationHooks.php: Backport: [[gerrit:783917{{!}}Hooks: return false rather than strings on failure (T305786)]] (duration: 01m 30s)
* 21:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25468 and previous config saved to /var/cache/conftool/dbconfig/20220419-214019-ladsgroup.json
* 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25467 and previous config saved to /var/cache/conftool/dbconfig/20220419-213707-ladsgroup.json
* 21:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 21:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 21:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25466 and previous config saved to /var/cache/conftool/dbconfig/20220419-213658-ladsgroup.json
* 21:25 ebernhardson: set index.unassigned.node_left.delayed_timeout to 10m for all indices in elasticsearch psi (:9200) cluster
* 21:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25465 and previous config saved to /var/cache/conftool/dbconfig/20220419-212514-ladsgroup.json
* 21:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25464 and previous config saved to /var/cache/conftool/dbconfig/20220419-212153-ladsgroup.json
* 21:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25463 and previous config saved to /var/cache/conftool/dbconfig/20220419-211824-ladsgroup.json
* 21:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 21:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 21:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25462 and previous config saved to /var/cache/conftool/dbconfig/20220419-211817-ladsgroup.json
* 21:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P25460 and previous config saved to /var/cache/conftool/dbconfig/20220419-210648-ladsgroup.json
* 21:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25459 and previous config saved to /var/cache/conftool/dbconfig/20220419-210311-ladsgroup.json
* 20:52 urbanecm: UTC late B&C window done
* 20:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25458 and previous config saved to /var/cache/conftool/dbconfig/20220419-205143-ladsgroup.json
* 20:49 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/extensions/GrowthExperiments/: {{Gerrit|e152df0}}: Revert "Skip welcome surveys for users in the no-homepage control group" ([[phab:T305015|T305015]]) (duration: 00m 55s)
* 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25457 and previous config saved to /var/cache/conftool/dbconfig/20220419-204826-ladsgroup.json
* 20:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 20:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25456 and previous config saved to /var/cache/conftool/dbconfig/20220419-204818-ladsgroup.json
* 20:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25455 and previous config saved to /var/cache/conftool/dbconfig/20220419-204806-ladsgroup.json
* 20:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:46 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25454 and previous config saved to /var/cache/conftool/dbconfig/20220419-203416-ladsgroup.json
* 20:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 20:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25453 and previous config saved to /var/cache/conftool/dbconfig/20220419-203313-ladsgroup.json
* 20:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25452 and previous config saved to /var/cache/conftool/dbconfig/20220419-203301-ladsgroup.json
* 20:31 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:27 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/includes/page/UndeletePage.php: {{Gerrit|f1ebd29}}: DeletePage, UndeletePage: use plaintextParams when creating log message ([[phab:T306431|T306431]]; 2/2) (duration: 00m 50s)
* 20:26 urbanecm@deploy1002: Synchronized php-1.39.0-wmf.8/includes/page/DeletePage.php: {{Gerrit|f1ebd29}}: DeletePage, UndeletePage: use plaintextParams when creating log message ([[phab:T306431|T306431]]; 1/2) (duration: 00m 50s)
* 20:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25451 and previous config saved to /var/cache/conftool/dbconfig/20220419-202618-ladsgroup.json
* 20:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 20:26 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 20:26 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25450 and previous config saved to /var/cache/conftool/dbconfig/20220419-202523-ladsgroup.json
* 20:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0a877710be56a06721d128868fd991b74e1f54a9}}: Add extendedconfirmed on elwiki ([[phab:T306241|T306241]]) (duration: 00m 50s)
* 20:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P25449 and previous config saved to /var/cache/conftool/dbconfig/20220419-201808-ladsgroup.json
* 20:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:10 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|f55f817}}: Add video marketing campaign to $wgGECampaignPattern ([[phab:T303785|T303785]]) (duration: 00m 54s)
* 20:10 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 20:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 20:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25448 and previous config saved to /var/cache/conftool/dbconfig/20220419-201018-ladsgroup.json
* 20:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25447 and previous config saved to /var/cache/conftool/dbconfig/20220419-200303-ladsgroup.json
* 19:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P25446 and previous config saved to /var/cache/conftool/dbconfig/20220419-195513-ladsgroup.json
* 19:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25445 and previous config saved to /var/cache/conftool/dbconfig/20220419-195050-ladsgroup.json
* 19:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
* 19:50 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudweb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 19:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 19:49 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 19:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 19:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
* 19:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25444 and previous config saved to /var/cache/conftool/dbconfig/20220419-194008-ladsgroup.json
* 19:40 urbanecm: [urbanecm@mwmaint1002 ~]$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/T304461.php --delete  # [[phab:T304461|T304461]]
* 19:35 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=frwiki --delete # [[phab:T304461|T304461]]
* 19:34 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=viwiki --delete # [[phab:T304461|T304461]]
* 19:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 19:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25443 and previous config saved to /var/cache/conftool/dbconfig/20220419-193318-ladsgroup.json
* 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25442 and previous config saved to /var/cache/conftool/dbconfig/20220419-193309-ladsgroup.json
* 19:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 19:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
* 19:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25441 and previous config saved to /var/cache/conftool/dbconfig/20220419-193301-ladsgroup.json
* 19:20 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudweb2002-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 19:20 bking@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - [[phab:T301959|T301959]]
* 19:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 19:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 19:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 19:20 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 19:19 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25440 and previous config saved to /var/cache/conftool/dbconfig/20220419-191812-ladsgroup.json
* 19:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25439 and previous config saved to /var/cache/conftool/dbconfig/20220419-191756-ladsgroup.json
* 19:15 jhuneidi@deploy1002: Pruned MediaWiki: 1.39.0-wmf.6 (duration: 01m 31s)
* 19:14 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 19:10 jhuneidi@deploy1002: Finished scap: testwikis wikis to 1.39.0-wmf.8  refs [[phab:T305214|T305214]] (duration: 42m 16s)
* 19:09 bking@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - [[phab:T301959|T301959]]
* 19:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P25438 and previous config saved to /var/cache/conftool/dbconfig/20220419-190306-ladsgroup.json
* 19:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P25437 and previous config saved to /var/cache/conftool/dbconfig/20220419-190250-ladsgroup.json
* 19:00 bking@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: Upgrading Elasticsearch to 6.8 in EQIAD - bking@cumin1001 - [[phab:T301959|T301959]]
* 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
* 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance
* 18:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 18:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
* 18:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25436 and previous config saved to /var/cache/conftool/dbconfig/20220419-185602-ladsgroup.json
* 18:53 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudservices2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 18:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25435 and previous config saved to /var/cache/conftool/dbconfig/20220419-184801-ladsgroup.json
* 18:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25434 and previous config saved to /var/cache/conftool/dbconfig/20220419-184745-ladsgroup.json
* 18:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25433 and previous config saved to /var/cache/conftool/dbconfig/20220419-184057-ladsgroup.json
* 18:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:39 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:39 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25432 and previous config saved to /var/cache/conftool/dbconfig/20220419-183544-ladsgroup.json
* 18:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 18:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
* 18:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25431 and previous config saved to /var/cache/conftool/dbconfig/20220419-183536-ladsgroup.json
* 18:34 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudservices2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 18:31 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudservices2004-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 18:27 jhuneidi@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.8  refs [[phab:T305214|T305214]]
* 18:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25430 and previous config saved to /var/cache/conftool/dbconfig/20220419-182552-ladsgroup.json
* 18:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25429 and previous config saved to /var/cache/conftool/dbconfig/20220419-182031-ladsgroup.json
* 18:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25428 and previous config saved to /var/cache/conftool/dbconfig/20220419-181047-ladsgroup.json
* 18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P25427 and previous config saved to /var/cache/conftool/dbconfig/20220419-180525-ladsgroup.json
* 18:05 brennen: train 1.38.0-wmf.9 ([[phab:T305214|T305214]]): we're currently debugging some scap / train prep issues.
* 18:04 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudservices2004-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 18:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 18:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 18:03 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudnet2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 17:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25426 and previous config saved to /var/cache/conftool/dbconfig/20220419-175431-ladsgroup.json
* 17:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 17:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 17:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25425 and previous config saved to /var/cache/conftool/dbconfig/20220419-175021-ladsgroup.json
* 17:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25424 and previous config saved to /var/cache/conftool/dbconfig/20220419-174731-ladsgroup.json
* 17:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:47 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 17:47 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 17:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25423 and previous config saved to /var/cache/conftool/dbconfig/20220419-174717-ladsgroup.json
* 17:41 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:39 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:39 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudnet2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 17:38 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:38 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25422 and previous config saved to /var/cache/conftool/dbconfig/20220419-173836-ladsgroup.json
* 17:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 17:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
* 17:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25421 and previous config saved to /var/cache/conftool/dbconfig/20220419-173827-ladsgroup.json
* 17:38 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudnet2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 17:38 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:38 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:38 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=arwiki --delete # [[phab:T304461|T304461]]
* 17:37 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25420 and previous config saved to /var/cache/conftool/dbconfig/20220419-173706-ladsgroup.json
* 17:36 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=bnwiki --delete # [[phab:T304461|T304461]]
* 17:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 17:33 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 17:33 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 17:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 17:33 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:33 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:32 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:32 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25419 and previous config saved to /var/cache/conftool/dbconfig/20220419-173212-ladsgroup.json
* 17:32 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:31 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25418 and previous config saved to /var/cache/conftool/dbconfig/20220419-172321-ladsgroup.json
* 17:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25417 and previous config saved to /var/cache/conftool/dbconfig/20220419-172200-ladsgroup.json
* 17:18 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P25416 and previous config saved to /var/cache/conftool/dbconfig/20220419-171707-ladsgroup.json
* 17:14 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 17:14 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:11 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudnet2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 17:11 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 17:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P25415 and previous config saved to /var/cache/conftool/dbconfig/20220419-170816-ladsgroup.json
* 17:07 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P25414 and previous config saved to /var/cache/conftool/dbconfig/20220419-170655-ladsgroup.json
* 17:02 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephmon2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25413 and previous config saved to /var/cache/conftool/dbconfig/20220419-170202-ladsgroup.json
* 16:56 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25412 and previous config saved to /var/cache/conftool/dbconfig/20220419-165641-kormat.json
* 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25411 and previous config saved to /var/cache/conftool/dbconfig/20220419-165511-ladsgroup.json
* 16:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 16:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 16:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25410 and previous config saved to /var/cache/conftool/dbconfig/20220419-165503-ladsgroup.json
* 16:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25409 and previous config saved to /var/cache/conftool/dbconfig/20220419-165311-ladsgroup.json
* 16:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 16:53 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 16:53 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 16:53 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 16:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25408 and previous config saved to /var/cache/conftool/dbconfig/20220419-165150-ladsgroup.json
* 16:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25407 and previous config saved to /var/cache/conftool/dbconfig/20220419-164216-ladsgroup.json
* 16:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 16:42 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephmon2006-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 16:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
* 16:41 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25406 and previous config saved to /var/cache/conftool/dbconfig/20220419-164137-kormat.json
* 16:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25405 and previous config saved to /var/cache/conftool/dbconfig/20220419-163958-ladsgroup.json
* 16:38 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25404 and previous config saved to /var/cache/conftool/dbconfig/20220419-163414-ladsgroup.json
* 16:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 16:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance
* 16:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25403 and previous config saved to /var/cache/conftool/dbconfig/20220419-163406-ladsgroup.json
* 16:33 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 16:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 16:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25402 and previous config saved to /var/cache/conftool/dbconfig/20220419-163321-ladsgroup.json
* 16:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2012.codfw.wmnet
* 16:32 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 16:31 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 16:28 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 16:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2012.codfw.wmnet
* 16:28 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 16:27 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 16:26 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25401 and previous config saved to /var/cache/conftool/dbconfig/20220419-162633-kormat.json
* 16:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P25400 and previous config saved to /var/cache/conftool/dbconfig/20220419-162453-ladsgroup.json
* 16:23 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=kowiki --delete # [[phab:T304461|T304461]]
* 16:21 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki=cswiki --delete # [[phab:T304461|T304461]]
* 16:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25399 and previous config saved to /var/cache/conftool/dbconfig/20220419-161901-ladsgroup.json
* 16:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25398 and previous config saved to /var/cache/conftool/dbconfig/20220419-161816-ladsgroup.json
* 16:16 otto@deploy1002: Finished deploy [analytics/refinery@f136555] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f136555] (duration: 06m 49s)
* 16:15 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 16:14 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 16:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephmon2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 16:11 kormat@cumin1001: dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25397 and previous config saved to /var/cache/conftool/dbconfig/20220419-161129-kormat.json
* 16:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25396 and previous config saved to /var/cache/conftool/dbconfig/20220419-160948-ladsgroup.json
* 16:09 otto@deploy1002: Started deploy [analytics/refinery@f136555] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f136555]
* 16:09 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1019.eqiad.wmnet with OS bullseye
* 16:08 otto@deploy1002: Finished deploy [analytics/refinery@f136555] (thin): Regular analytics weekly train THIN [analytics/refinery@f136555] (duration: 00m 07s)
* 16:08 otto@deploy1002: Started deploy [analytics/refinery@f136555] (thin): Regular analytics weekly train THIN [analytics/refinery@f136555]
* 16:07 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 16:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 16:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 16:06 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25395 and previous config saved to /var/cache/conftool/dbconfig/20220419-160629-kormat.json
* 16:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25394 and previous config saved to /var/cache/conftool/dbconfig/20220419-160409-ladsgroup.json
* 16:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 16:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance
* 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P25393 and previous config saved to /var/cache/conftool/dbconfig/20220419-160355-ladsgroup.json
* 16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P25392 and previous config saved to /var/cache/conftool/dbconfig/20220419-160311-ladsgroup.json
* 15:59 otto@deploy1002: Finished deploy [analytics/refinery@f136555]: weekly train (duration: 22m 21s)
* 15:57 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:57 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:57 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:55 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25391 and previous config saved to /var/cache/conftool/dbconfig/20220419-155531-kormat.json
* 15:54 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:54 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:54 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:54 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:51 kormat@cumin1001: dbctl commit (dc=all): 'es1026 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25390 and previous config saved to /var/cache/conftool/dbconfig/20220419-155146-kormat.json
* 15:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1026.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:51 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1026.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:51 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25389 and previous config saved to /var/cache/conftool/dbconfig/20220419-155125-kormat.json
* 15:51 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:51 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:50 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:50 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:50 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1019.eqiad.wmnet with reason: host reimage
* 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25388 and previous config saved to /var/cache/conftool/dbconfig/20220419-154850-ladsgroup.json
* 15:48 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudcephmon2005-dev.mgmt.codfw.wmnet with reboot policy FORCED
* 15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25387 and previous config saved to /var/cache/conftool/dbconfig/20220419-154806-ladsgroup.json
* 15:47 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1019.eqiad.wmnet with reason: host reimage
* 15:46 damilare: payments-wiki revision changed from {{Gerrit|a9a1f2ee}} to {{Gerrit|a3c69385}}
* 15:45 damilare: localsettings revision changed from {{Gerrit|c8fee00c}} to {{Gerrit|e365fe0a}}
* 15:40 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25386 and previous config saved to /var/cache/conftool/dbconfig/20220419-154027-kormat.json
* 15:39 elukey: powercycle elastic1097 (still with role::insetup, but not reachable via ssh or mgmt console)
* 15:37 otto@deploy1002: Started deploy [analytics/refinery@f136555]: weekly train
* 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25385 and previous config saved to /var/cache/conftool/dbconfig/20220419-153707-ladsgroup.json
* 15:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 15:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
* 15:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25384 and previous config saved to /var/cache/conftool/dbconfig/20220419-153659-ladsgroup.json
* 15:36 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2008.codfw.wmnet with OS bullseye
* 15:36 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25383 and previous config saved to /var/cache/conftool/dbconfig/20220419-153621-kormat.json
* 15:35 ariel@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host dumpsdata1003.eqiad.wmnet
* 15:35 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1019.eqiad.wmnet with OS bullseye
* 15:33 elukey: start rdb2008 from mgmt console (was powered down for relocation)
* 15:29 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host wdqs2011.codfw.wmnet
* 15:28 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:27 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2007.codfw.wmnet with OS bullseye
* 15:26 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:26 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:26 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:25 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1003.eqiad.wmnet
* 15:25 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25382 and previous config saved to /var/cache/conftool/dbconfig/20220419-152523-kormat.json
* 15:25 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:25 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:24 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2006.codfw.wmnet with OS bullseye
* 15:24 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:24 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2008.codfw.wmnet with reason: host reimage
* 15:24 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:24 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:24 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 15:24 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:24 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:23 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 15:23 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 15:22 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet
* 15:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25381 and previous config saved to /var/cache/conftool/dbconfig/20220419-152154-ladsgroup.json
* 15:21 kormat@cumin1001: dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25380 and previous config saved to /var/cache/conftool/dbconfig/20220419-152117-kormat.json
* 15:19 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2008.codfw.wmnet with reason: host reimage
* 15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25379 and previous config saved to /var/cache/conftool/dbconfig/20220419-151847-ladsgroup.json
* 15:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2011.codfw.wmnet
* 15:17 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host wdqs2010.codfw.wmnet
* 15:17 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet
* 15:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1130 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25378 and previous config saved to /var/cache/conftool/dbconfig/20220419-151607-ladsgroup.json
* 15:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 15:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1130.eqiad.wmnet with reason: Maintenance
* 15:15 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2007.codfw.wmnet with reason: host reimage
* 15:15 kormat@cumin1001: dbctl commit (dc=all): 'es1027 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25377 and previous config saved to /var/cache/conftool/dbconfig/20220419-151552-kormat.json
* 15:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on es1027.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on es1027.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:13 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2006.codfw.wmnet with reason: host reimage
* 15:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
* 15:10 kormat@cumin1001: dbctl commit (dc=all): 'db1114 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25376 and previous config saved to /var/cache/conftool/dbconfig/20220419-151019-kormat.json
* 15:10 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2005.codfw.wmnet with OS bullseye
* 15:10 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2007.codfw.wmnet with reason: host reimage
* 15:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 15:09 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2006.codfw.wmnet with reason: host reimage
* 15:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 15:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 15:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2123.codfw.wmnet with reason: Maintenance
* 15:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2010.codfw.wmnet
* 15:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2009.codfw.wmnet
* 15:07 kormat@cumin1001: dbctl commit (dc=all): 'db1182 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25375 and previous config saved to /var/cache/conftool/dbconfig/20220419-150717-kormat.json
* 15:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1182.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P25374 and previous config saved to /var/cache/conftool/dbconfig/20220419-150649-ladsgroup.json
* 15:06 kormat@cumin1001: dbctl commit (dc=all): 'db1114 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25373 and previous config saved to /var/cache/conftool/dbconfig/20220419-150637-kormat.json
* 15:06 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1114.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:06 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1114.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 15:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25372 and previous config saved to /var/cache/conftool/dbconfig/20220419-150454-ladsgroup.json
* 15:03 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2008.codfw.wmnet with OS bullseye
* 15:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host wdqs2009.codfw.wmnet
* 15:03 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2004.codfw.wmnet with OS bullseye
* 14:58 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2005.codfw.wmnet with reason: host reimage
* 14:56 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25371 and previous config saved to /var/cache/conftool/dbconfig/20220419-145658-kormat.json
* 14:54 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2005.codfw.wmnet with reason: host reimage
* 14:54 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2007.codfw.wmnet with OS bullseye
* 14:54 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2006.codfw.wmnet with OS bullseye
* 14:52 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2003.codfw.wmnet with OS bullseye
* 14:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25370 and previous config saved to /var/cache/conftool/dbconfig/20220419-145143-ladsgroup.json
* 14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25369 and previous config saved to /var/cache/conftool/dbconfig/20220419-144949-ladsgroup.json
* 14:49 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25368 and previous config saved to /var/cache/conftool/dbconfig/20220419-144941-kormat.json
* 14:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25367 and previous config saved to /var/cache/conftool/dbconfig/20220419-144836-ladsgroup.json
* 14:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 14:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 14:48 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2002.codfw.wmnet with OS bullseye
* 14:45 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: host reimage
* 14:42 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2004.codfw.wmnet with reason: host reimage
* 14:41 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25366 and previous config saved to /var/cache/conftool/dbconfig/20220419-144154-kormat.json
* 14:41 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25365 and previous config saved to /var/cache/conftool/dbconfig/20220419-144144-kormat.json
* 14:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25364 and previous config saved to /var/cache/conftool/dbconfig/20220419-144105-ladsgroup.json
* 14:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 14:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
* 14:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25363 and previous config saved to /var/cache/conftool/dbconfig/20220419-144057-ladsgroup.json
* 14:40 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25362 and previous config saved to /var/cache/conftool/dbconfig/20220419-144001-kormat.json
* 14:39 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2003.codfw.wmnet with reason: host reimage
* 14:39 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2005.codfw.wmnet with OS bullseye
* 14:38 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve2001.codfw.wmnet with OS bullseye
* 14:36 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2002.codfw.wmnet with reason: host reimage
* 14:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P25361 and previous config saved to /var/cache/conftool/dbconfig/20220419-143444-ladsgroup.json
* 14:34 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25360 and previous config saved to /var/cache/conftool/dbconfig/20220419-143437-kormat.json
* 14:33 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2003.codfw.wmnet with reason: host reimage
* 14:33 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: host reimage
* 14:26 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25359 and previous config saved to /var/cache/conftool/dbconfig/20220419-142650-kormat.json
* 14:26 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25358 and previous config saved to /var/cache/conftool/dbconfig/20220419-142640-kormat.json
* 14:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25357 and previous config saved to /var/cache/conftool/dbconfig/20220419-142552-ladsgroup.json
* 14:25 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2001.codfw.wmnet with reason: host reimage
* 14:25 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2004.codfw.wmnet with OS bullseye
* 14:24 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25356 and previous config saved to /var/cache/conftool/dbconfig/20220419-142457-kormat.json
* 14:22 elukey@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2001.codfw.wmnet with reason: host reimage
* 14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25355 and previous config saved to /var/cache/conftool/dbconfig/20220419-141937-ladsgroup.json
* 14:19 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25354 and previous config saved to /var/cache/conftool/dbconfig/20220419-141933-kormat.json
* 14:17 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2003.codfw.wmnet with OS bullseye
* 14:16 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2002.codfw.wmnet with OS bullseye
* 14:15 jynus: edited directly phab database to fix corrupt entry [[phab:T305919|T305919]]
* 14:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25353 and previous config saved to /var/cache/conftool/dbconfig/20220419-141303-ladsgroup.json
* 14:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 14:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1110.eqiad.wmnet with reason: Maintenance
* 14:11 kormat@cumin1001: dbctl commit (dc=all): 'db1111 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25352 and previous config saved to /var/cache/conftool/dbconfig/20220419-141146-kormat.json
* 14:11 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25351 and previous config saved to /var/cache/conftool/dbconfig/20220419-141136-kormat.json
* 14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P25350 and previous config saved to /var/cache/conftool/dbconfig/20220419-141047-ladsgroup.json
* 14:09 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25349 and previous config saved to /var/cache/conftool/dbconfig/20220419-140954-kormat.json
* 14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1111 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25348 and previous config saved to /var/cache/conftool/dbconfig/20220419-140756-kormat.json
* 14:07 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1111.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:07 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1111.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:07 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 100%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25347 and previous config saved to /var/cache/conftool/dbconfig/20220419-140703-kormat.json
* 14:06 godog: start deleting tegola-cache/osm prefix from tegola-swift-container - [[phab:T306424|T306424]]
* 14:05 elukey@cumin1001: START - Cookbook sre.hosts.reimage for host ml-serve2001.codfw.wmnet with OS bullseye
* 14:04 kormat@cumin1001: dbctl commit (dc=all): 'db1129 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25346 and previous config saved to /var/cache/conftool/dbconfig/20220419-140430-kormat.json
* 14:01 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 14:01 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:56 kormat@cumin1001: dbctl commit (dc=all): 'db1110 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25345 and previous config saved to /var/cache/conftool/dbconfig/20220419-135632-kormat.json
* 13:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25344 and previous config saved to /var/cache/conftool/dbconfig/20220419-135542-ladsgroup.json
* 13:55 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
* 13:54 kormat@cumin1001: dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25343 and previous config saved to /var/cache/conftool/dbconfig/20220419-135450-kormat.json
* 13:52 kormat@cumin1001: dbctl commit (dc=all): 'db1110 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25342 and previous config saved to /var/cache/conftool/dbconfig/20220419-135225-kormat.json
* 13:52 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1110.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:52 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1110.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:51 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 75%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25341 and previous config saved to /var/cache/conftool/dbconfig/20220419-135159-kormat.json
* 13:51 kormat@cumin1001: dbctl commit (dc=all): 'db1129 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25340 and previous config saved to /var/cache/conftool/dbconfig/20220419-135140-kormat.json
* 13:51 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:51 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1129.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:50 kormat@cumin1001: dbctl commit (dc=all): 'db1169 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25339 and previous config saved to /var/cache/conftool/dbconfig/20220419-135007-kormat.json
* 13:50 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1169.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:50 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1169.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:46 hnowlan@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad
* 13:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25338 and previous config saved to /var/cache/conftool/dbconfig/20220419-134503-ladsgroup.json
* 13:45 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 13:45 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
* 13:44 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25337 and previous config saved to /var/cache/conftool/dbconfig/20220419-134455-ladsgroup.json
* 13:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25336 and previous config saved to /var/cache/conftool/dbconfig/20220419-134139-marostegui.json
* 13:36 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 50%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25335 and previous config saved to /var/cache/conftool/dbconfig/20220419-133655-kormat.json
* 13:30 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db2080.codfw.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:30 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db2080.codfw.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:29 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25334 and previous config saved to /var/cache/conftool/dbconfig/20220419-132949-ladsgroup.json
* 13:27 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784255{{!}}mrwikisource: Add template editor and patroller user groups (T269067)]] (duration: 00m 50s)
* 13:27 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 13:26 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 13:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25333 and previous config saved to /var/cache/conftool/dbconfig/20220419-132634-marostegui.json
* 13:26 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
* 13:25 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
* 13:25 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 13:25 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 13:25 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 13:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 13:21 kormat@cumin1001: dbctl commit (dc=all): 'db1104 (re)pooling @ 25%: Reboot [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25332 and previous config saved to /var/cache/conftool/dbconfig/20220419-132151-kormat.json
* 13:15 kormat@cumin1001: dbctl commit (dc=all): 'db1104 depooling: Rebooting for [[phab:T303174|T303174]]', diff saved to https://phabricator.wikimedia.org/P25331 and previous config saved to /var/cache/conftool/dbconfig/20220419-131557-kormat.json
* 13:15 kormat@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1104.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:15 kormat@cumin1001: START - Cookbook sre.hosts.downtime for 1:30:00 on db1104.eqiad.wmnet with reason: Rebooting for [[phab:T303174|T303174]]
* 13:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P25330 and previous config saved to /var/cache/conftool/dbconfig/20220419-131444-ladsgroup.json
* 13:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P25329 and previous config saved to /var/cache/conftool/dbconfig/20220419-131128-marostegui.json
* 13:03 volans@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 13:03 volans@cumin1001: START - Cookbook sre.network.cf
* 12:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25328 and previous config saved to /var/cache/conftool/dbconfig/20220419-125939-ladsgroup.json
* 12:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25327 and previous config saved to /var/cache/conftool/dbconfig/20220419-125623-marostegui.json
* 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25326 and previous config saved to /var/cache/conftool/dbconfig/20220419-124851-ladsgroup.json
* 12:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 12:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
* 12:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25325 and previous config saved to /var/cache/conftool/dbconfig/20220419-124843-ladsgroup.json
* 12:47 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 12:46 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 12:46 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 12:46 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
* 12:45 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
* 12:41 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 12:41 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 12:40 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 12:38 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 12:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25324 and previous config saved to /var/cache/conftool/dbconfig/20220419-123337-ladsgroup.json
* 12:31 mmandere@cumin1001: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for pybal-test2002.codfw.wmnet: Renew puppet certificate - mmandere@cumin1001
* 12:31 mmandere@cumin1001: START - Cookbook sre.puppet.renew-cert for pybal-test2002.codfw.wmnet: Renew puppet certificate - mmandere@cumin1001
* 12:23 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
* 12:22 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
* 12:21 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
* 12:20 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
* 12:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P25323 and previous config saved to /var/cache/conftool/dbconfig/20220419-121832-ladsgroup.json
* 12:16 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host db1136.eqiad.wmnet with OS bullseye
* 12:14 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
* 12:12 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
* 12:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25322 and previous config saved to /var/cache/conftool/dbconfig/20220419-120327-ladsgroup.json
* 12:02 godog: create tegola-swift-fallback container in account tegola
* 12:01 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1136.eqiad.wmnet with reason: host reimage
* 11:57 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1136.eqiad.wmnet with reason: host reimage
* 11:56 hnowlan@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw
* 11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1180 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25321 and previous config saved to /var/cache/conftool/dbconfig/20220419-115609-marostegui.json
* 11:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 11:56 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25320 and previous config saved to /var/cache/conftool/dbconfig/20220419-115601-marostegui.json
* 11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25319 and previous config saved to /var/cache/conftool/dbconfig/20220419-115239-ladsgroup.json
* 11:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 11:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 11:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 11:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance
* 11:47 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1136.eqiad.wmnet with OS bullseye
* 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
* 11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
* 11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25318 and previous config saved to /var/cache/conftool/dbconfig/20220419-114311-ladsgroup.json
* 11:40 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25317 and previous config saved to /var/cache/conftool/dbconfig/20220419-114056-marostegui.json
* 11:32 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 11:30 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 11:28 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 11:28 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 11:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25316 and previous config saved to /var/cache/conftool/dbconfig/20220419-112806-ladsgroup.json
* 11:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P25315 and previous config saved to /var/cache/conftool/dbconfig/20220419-112551-marostegui.json
* 11:25 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:25 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:24 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 11:23 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 11:21 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
* 11:21 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
* 11:18 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:18 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P25314 and previous config saved to /var/cache/conftool/dbconfig/20220419-111301-ladsgroup.json
* 11:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25313 and previous config saved to /var/cache/conftool/dbconfig/20220419-111046-marostegui.json
* 11:10 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
* 11:09 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
* 11:08 hnowlan@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'similar-users' for release 'main' .
* 11:07 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:07 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25312 and previous config saved to /var/cache/conftool/dbconfig/20220419-110710-marostegui.json
* 11:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 11:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 11:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:05 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:05 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 11:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
* 11:04 moritzm: installing xz-utils/xzgrep security updates
* 11:04 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:03 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:02 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 11:02 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance
* 11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance
* 11:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 11:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance
* 10:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 10:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
* 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25311 and previous config saved to /var/cache/conftool/dbconfig/20220419-105948-marostegui.json
* 10:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25310 and previous config saved to /var/cache/conftool/dbconfig/20220419-105756-ladsgroup.json
* 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25309 and previous config saved to /var/cache/conftool/dbconfig/20220419-104443-marostegui.json
* 10:39 mmandere: reimage pybal-test2002 as buster - [[phab:T297187|T297187]]
* 10:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P25308 and previous config saved to /var/cache/conftool/dbconfig/20220419-102938-marostegui.json
* 10:17 moritzm: installing gzip/zgrep security updates
* 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25306 and previous config saved to /var/cache/conftool/dbconfig/20220419-101433-marostegui.json
* 10:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1131 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25305 and previous config saved to /var/cache/conftool/dbconfig/20220419-101233-marostegui.json
* 10:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 10:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
* 10:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 ([[phab:T306269|T306269]])', diff saved to https://phabricator.wikimedia.org/P25304 and previous config saved to /var/cache/conftool/dbconfig/20220419-101225-marostegui.json
* 09:57 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25303 and previous config saved to /var/cache/conftool/dbconfig/20220419-095742-ladsgroup.json
* 09:57 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 09:57 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
* 09:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25302 and previous config saved to /var/cache/conftool/dbconfig/20220419-095720-marostegui.json
* 09:50 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 09:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
* 09:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
* 09:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 ([[phab:T298565|T298565]])', diff saved to https://phabricator.wikimedia.org/P25301 and previous config saved to /var/cache/conftool/dbconfig/20220419-094812-ladsgroup.json
* 09:43 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
* 09:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P25300 and previous config saved to /var/cache/conftool/dbconfig/20220419-094215-marostegui.json