Server Admin Log: Difference between revisions

Revision as of 00:15, 25 May 2022

2022-05-25

00:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28462 and previous config saved to /var/cache/conftool/dbconfig/20220525-001552-ladsgroup.json
00:15 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance
00:15 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1096.eqiad.wmnet with reason: Maintenance

2022-05-24

22:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
22:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
22:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
22:04 cjming: end of UTC late backport window
22:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
22:03 mutante: centrallog2002 - alerted because running out of disk. /srv/syslog# find . -name *.gz -mtime +100 -delete
22:02 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki" (T233004 T309148) (duration: 00m 49s)
21:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:58 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:56 cjming@deploy1002: Synchronized php-1.39.0-wmf.13/extensions/MobileFrontend: Backport: Follow-up I97c27fd7: Fix after-edit reload in source editor (T309068) (duration: 00m 48s)
21:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:41 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:41 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:36 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings-labs.php: Config: Update beta cluster DiscussionTools A/B test config (T304030) (duration: 00m 49s)
21:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:35 cjming@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Disable autotopicsub user option by default (T297966) (duration: 00m 48s)
21:34 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
21:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:30 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:27 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: zhwikisource: Adjust workmark size (T308620) (duration: 00m 50s)
21:26 cjming@deploy1002: Synchronized static/images/mobile/copyright/wikisource-wordmark-zh.svg: Config: zhwikisource: Adjust workmark size (T308620) (duration: 00m 50s)
21:23 cjming@deploy1002: Synchronized php-1.39.0-wmf.12/resources/src/mediawiki.skinning/accessibility.less: Backport: mediawiki.skinning: `transition-duration` accessibility override set to `0` (T308979) (duration: 00m 51s)
21:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:06 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Start writing to cuc_actor in s3, kcgwiki and labtestwiki (T233004) (duration: 00m 52s)
21:05 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
21:00 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Deploy IPInfo to all wikis by default (T260597) (duration: 00m 52s)
20:58 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:54 cjming@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Add comment to consult Legal before updating IPInfo access (T308876) (duration: 00m 52s)
20:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:20 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host relforge1003.eqiad.wmnet with OS bullseye
20:15 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:15 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:08 cjming@deploy1002: Synchronized wmf-config/CommonSettings-labs.php: Config: Remove outdated comment about IPInfo from CommonSettings-labs.php (T308876) (duration: 00m 49s)
20:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:02 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1003.eqiad.wmnet with reason: host reimage
19:59 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1003.eqiad.wmnet with reason: host reimage
19:53 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
19:49 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host relforge1003.eqiad.wmnet with OS bullseye
19:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
19:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
19:43 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
19:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
19:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
19:31 ryankemper@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:30 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:29 mforns@deploy1002: Finished deploy [airflow-dags/analytics@3ae51e7]: (no justification provided) (duration: 00m 08s)
19:29 mforns@deploy1002: Started deploy [airflow-dags/analytics@3ae51e7]: (no justification provided)
19:24 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:24 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:24 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:24 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:22 ebysans@deploy1002: Finished deploy [analytics/refinery@8314d31] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@8314d31] (duration: 07m 21s)
19:22 ryankemper@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:21 ryankemper@cumin1001: START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reimage - ryankemper@cumin1001 - T308606
19:15 ebysans@deploy1002: Started deploy [analytics/refinery@8314d31] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@8314d31]
19:14 ebysans@deploy1002: Finished deploy [analytics/refinery@8314d31] (thin): Regular analytics weekly train THIN [analytics/refinery@8314d31] (duration: 00m 08s)
19:14 ebysans@deploy1002: Started deploy [analytics/refinery@8314d31] (thin): Regular analytics weekly train THIN [analytics/refinery@8314d31]
19:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
19:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
19:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
19:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
19:07 dancy@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.13 refs T305219
19:01 ebysans@deploy1002: Finished deploy [analytics/refinery@8314d31]: Regular analytics weekly train [analytics/refinery@8314d31] (duration: 23m 40s)
18:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:48 dancy@deploy1002: Pruned MediaWiki: 1.39.0-wmf.9, 1.39.0-wmf.8, 1.39.0-wmf.10 (duration: 02m 28s)
18:46 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:46 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:45 dancy@deploy1002: Finished scap: testwikis wikis to 1.39.0-wmf.13 refs T305219 (duration: 41m 45s)
18:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:38 ebysans@deploy1002: Started deploy [analytics/refinery@8314d31]: Regular analytics weekly train [analytics/refinery@8314d31]
18:36 SandraEbele: deploying analytics refinery as part of the weekly deployment
18:11 robh: cp6006 memory issue resolved, returned system to service and ended maint window via T309123
18:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
18:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:03 dancy@deploy1002: Started scap: testwikis wikis to 1.39.0-wmf.13 refs T305219
18:03 robh: cp6006 in maint mode and depooled for memory troubleshooting via T309123
17:53 aokoth@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2003.wikimedia.org with OS bullseye
17:44 cmooney@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
17:41 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:39 aokoth@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage
17:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:37 aokoth@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage
17:35 ladsgroup@deploy1002: Synchronized php-1.39.0-wmf.12/includes/api/ApiQueryBacklinksprop.php: Backport: Revert "ApiQueryBacklinksprop: Completely remove index hints" (duration: 00m 50s)
17:25 cmooney@cumin1001: START - Cookbook sre.dns.netbox
17:23 mutante: gitlab1003 - short downtime for maintenance
17:21 aokoth@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye
17:18 moritzm: failover ganeti master in codfw to ganeti2022
17:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
17:17 aokoth@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2002.wikimedia.org with OS bullseye
17:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 8 hosts with reason: Maintenance
17:17 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
17:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance
17:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298560)', diff saved to https://phabricator.wikimedia.org/P28459 and previous config saved to /var/cache/conftool/dbconfig/20220524-171736-ladsgroup.json
17:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet
17:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet
17:04 aokoth@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage
17:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P28457 and previous config saved to /var/cache/conftool/dbconfig/20220524-170231-ladsgroup.json
17:00 aokoth@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage
16:50 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab1003.wikimedia.org with reason: fsck
16:50 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab1003.wikimedia.org with reason: fsck
16:50 mutante: gitlab1003 (gitlab-replica-new) - rebooting for fsck - T307142
16:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P28456 and previous config saved to /var/cache/conftool/dbconfig/20220524-164726-ladsgroup.json
16:45 aokoth@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye
16:33 mutante: gitlab1003 - restarting rsync, trying to debug mysterious "rsync - read-only file system" error we ran into before but could not reproduce
16:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T298560)', diff saved to https://phabricator.wikimedia.org/P28455 and previous config saved to /var/cache/conftool/dbconfig/20220524-163221-ladsgroup.json
15:31 hnowlan@deploy1002: helmfile [staging] DONE helmfile.d/services/image-suggestion: apply
15:27 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@644075e]: increase executor jvm heap for convert_to_esbulk (duration: 02m 22s)
15:27 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:25 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@644075e]: increase executor jvm heap for convert_to_esbulk
15:22 volans@cumin1001: START - Cookbook sre.dns.netbox
15:21 hnowlan@deploy1002: helmfile [staging] START helmfile.d/services/image-suggestion: apply
15:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet
15:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet
14:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet
14:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet
14:03 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2068.codfw.wmnet
14:02 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:55 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Revert "Revert read new on frwiki for templatelinks migration" (duration: 00m 52s)
13:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:52 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:52 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:51 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:50 ladsgroup@deploy1002: Synchronized php-1.39.0-wmf.12/includes/api/ApiQueryBacklinksprop.php: Backport: ApiQueryBacklinksprop: Completely remove index hints (T306673) (duration: 00m 55s)
13:42 mvernon@cumin2002: START - Cookbook sre.hosts.reboot-single for host ms-be2068.codfw.wmnet
13:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:28 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:27 taavi@deploy1002: Synchronized static/images/project-logos: Config: zhwikisource: Optimize logo per commons files (T308620) (duration: 00m 55s)
13:27 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:25 taavi@deploy1002: Synchronized wmf-config/logos.php: Config: zhwikisource: Declare commons files for logo (T308620) (duration: 00m 52s)
13:25 taavi@deploy1002: Synchronized logos/config.yaml: Config: zhwikisource: Declare commons files for logo (T308620) (duration: 00m 53s)
13:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:13 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:10 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Remove patrol rights from autoconfirmed users and create patroller user group on bnwiki (T308945) (duration: 00m 53s)
12:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T303603)', diff saved to https://phabricator.wikimedia.org/P28452 and previous config saved to /var/cache/conftool/dbconfig/20220524-125331-ladsgroup.json
12:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet
12:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet
12:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P28450 and previous config saved to /var/cache/conftool/dbconfig/20220524-123826-ladsgroup.json
12:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet
12:31 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet
12:30 moritzm: installing openldap security updates
12:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P28449 and previous config saved to /var/cache/conftool/dbconfig/20220524-122321-ladsgroup.json
12:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet
12:17 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet
12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T298560)', diff saved to https://phabricator.wikimedia.org/P28448 and previous config saved to /var/cache/conftool/dbconfig/20220524-121641-ladsgroup.json
12:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
12:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
12:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
12:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
12:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28447 and previous config saved to /var/cache/conftool/dbconfig/20220524-121627-ladsgroup.json
12:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1118 (T303603)', diff saved to https://phabricator.wikimedia.org/P28446 and previous config saved to /var/cache/conftool/dbconfig/20220524-120816-ladsgroup.json
12:06 jbond: disable puppet on c:httpd
12:04 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2024.codfw.wmnet
12:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P28445 and previous config saved to /var/cache/conftool/dbconfig/20220524-120122-ladsgroup.json
11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1118 (T303603)', diff saved to https://phabricator.wikimedia.org/P28444 and previous config saved to /var/cache/conftool/dbconfig/20220524-115251-ladsgroup.json
11:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1118.eqiad.wmnet with reason: Maintenance
11:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1118.eqiad.wmnet with reason: Maintenance
11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T303603)', diff saved to https://phabricator.wikimedia.org/P28443 and previous config saved to /var/cache/conftool/dbconfig/20220524-115243-ladsgroup.json
11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P28442 and previous config saved to /var/cache/conftool/dbconfig/20220524-114617-ladsgroup.json
11:45 jbond: disable puppet on mw servers
11:40 jbond@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host puppetmaster1001.eqiad.wmnet
11:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P28441 and previous config saved to /var/cache/conftool/dbconfig/20220524-113738-ladsgroup.json
11:34 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetmaster1001.eqiad.wmnet
11:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2024.codfw.wmnet
11:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28440 and previous config saved to /var/cache/conftool/dbconfig/20220524-113112-ladsgroup.json
11:30 jbond: disable puppet fleet wide
11:23 elukey@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P28439 and previous config saved to /var/cache/conftool/dbconfig/20220524-112233-ladsgroup.json
11:19 elukey@cumin1001: START - Cookbook sre.dns.netbox
11:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2023.codfw.wmnet
11:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2023.codfw.wmnet
11:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T303603)', diff saved to https://phabricator.wikimedia.org/P28438 and previous config saved to /var/cache/conftool/dbconfig/20220524-110728-ladsgroup.json
11:00 jynus: restart db1150 T308315
10:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T303603)', diff saved to https://phabricator.wikimedia.org/P28437 and previous config saved to /var/cache/conftool/dbconfig/20220524-105116-ladsgroup.json
10:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
10:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
10:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T303603)', diff saved to https://phabricator.wikimedia.org/P28436 and previous config saved to /var/cache/conftool/dbconfig/20220524-105108-ladsgroup.json
10:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet
10:43 jbond@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2005.codfw.wmnet
10:43 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2004.codfw.wmnet
10:39 jbond@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2005.codfw.wmnet
10:38 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetmaster2004.codfw.wmnet
10:38 jbond@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2003.codfw.wmnet
10:38 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster1005.eqiad.wmnet
10:37 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster1003.eqiad.wmnet
10:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P28435 and previous config saved to /var/cache/conftool/dbconfig/20220524-103603-ladsgroup.json
10:33 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetmaster1003.eqiad.wmnet
10:33 jbond@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2003.codfw.wmnet
10:33 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster1002.eqiad.wmnet
10:32 jbond@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster2002.codfw.wmnet
10:32 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetmaster1005.eqiad.wmnet
10:32 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetmaster1004.eqiad.wmnet
10:28 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetmaster1004.eqiad.wmnet
10:28 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb2002.codfw.wmnet
10:28 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetmaster1002.eqiad.wmnet
10:27 jbond@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2002.codfw.wmnet
10:26 jbond@cumin2002: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host puppetmaster2001.codfw.wmnet
10:26 jbond@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb1002.eqiad.wmnet
10:25 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetdb2002.codfw.wmnet
10:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P28434 and previous config saved to /var/cache/conftool/dbconfig/20220524-102058-ladsgroup.json
10:20 moritzm: installing vim security updates
10:18 moritzm: rebalance Ganeti cluster in eqsin T308211
10:15 jbond@cumin2002: START - Cookbook sre.hosts.reboot-single for host puppetmaster2001.codfw.wmnet
10:14 jbond@cumin1001: START - Cookbook sre.hosts.reboot-single for host puppetdb1002.eqiad.wmnet
10:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2022.codfw.wmnet
10:07 moritzm: installing imagemagick securitx updates
10:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T303603)', diff saved to https://phabricator.wikimedia.org/P28432 and previous config saved to /var/cache/conftool/dbconfig/20220524-100553-ladsgroup.json
09:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet
09:51 btullis@cumin1001: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:aqs: Rolling AQS Cassandra cluster to pick up new encryption settings - btullis@cumin1001
09:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T303603)', diff saved to https://phabricator.wikimedia.org/P28431 and previous config saved to /var/cache/conftool/dbconfig/20220524-095030-ladsgroup.json
09:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
09:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
09:50 moritzm: installing openssl security updates
09:49 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet
09:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T303603)', diff saved to https://phabricator.wikimedia.org/P28430 and previous config saved to /var/cache/conftool/dbconfig/20220524-093830-ladsgroup.json
09:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2017.codfw.wmnet
09:34 root@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti5001.eqsin.wmnet to ganeti01.svc.eqsin.wmnet
09:33 root@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5001.eqsin.wmnet to ganeti01.svc.eqsin.wmnet
09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5001.eqsin.wmnet
09:28 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2017.codfw.wmnet
09:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P28428 and previous config saved to /var/cache/conftool/dbconfig/20220524-092324-ladsgroup.json
09:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5001.eqsin.wmnet
09:13 btullis@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs: Rolling AQS Cassandra cluster to pick up new encryption settings - btullis@cumin1001
09:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P28427 and previous config saved to /var/cache/conftool/dbconfig/20220524-090819-ladsgroup.json
09:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2016.codfw.wmnet
09:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2016.codfw.wmnet
08:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T303603)', diff saved to https://phabricator.wikimedia.org/P28425 and previous config saved to /var/cache/conftool/dbconfig/20220524-085314-ladsgroup.json
08:44 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
08:43 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
08:43 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
08:43 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
08:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2015.codfw.wmnet
08:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T303603)', diff saved to https://phabricator.wikimedia.org/P28424 and previous config saved to /var/cache/conftool/dbconfig/20220524-083822-ladsgroup.json
08:38 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
08:38 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
08:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T303603)', diff saved to https://phabricator.wikimedia.org/P28423 and previous config saved to /var/cache/conftool/dbconfig/20220524-083814-ladsgroup.json
08:33 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
08:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2015.codfw.wmnet
08:33 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
08:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P28422 and previous config saved to /var/cache/conftool/dbconfig/20220524-082309-ladsgroup.json
08:22 godog: resume deletion of 'swift-tegola-container' on thanos-fe2001 - T307184
08:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2014.codfw.wmnet
08:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2014.codfw.wmnet
08:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P28421 and previous config saved to /var/cache/conftool/dbconfig/20220524-080804-ladsgroup.json
08:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298555)', diff saved to https://phabricator.wikimedia.org/P28420 and previous config saved to /var/cache/conftool/dbconfig/20220524-080758-ladsgroup.json
07:57 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:57 oblivian@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:56 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:56 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2013.codfw.wmnet
07:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T303603)', diff saved to https://phabricator.wikimedia.org/P28419 and previous config saved to /var/cache/conftool/dbconfig/20220524-075259-ladsgroup.json
07:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P28418 and previous config saved to /var/cache/conftool/dbconfig/20220524-075253-ladsgroup.json
07:52 oblivian@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:52 oblivian@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:48 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2013.codfw.wmnet
07:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298555)', diff saved to https://phabricator.wikimedia.org/P28417 and previous config saved to /var/cache/conftool/dbconfig/20220524-074607-ladsgroup.json
07:38 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P28416 and previous config saved to /var/cache/conftool/dbconfig/20220524-073748-ladsgroup.json
07:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T303603)', diff saved to https://phabricator.wikimedia.org/P28415 and previous config saved to /var/cache/conftool/dbconfig/20220524-073738-ladsgroup.json
07:37 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
07:37 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance
07:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:35 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:33 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Remove wgPriorityHints and wgPriorityHintsRatio (T308707) (duration: 00m 50s)
07:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P28414 and previous config saved to /var/cache/conftool/dbconfig/20220524-073102-ladsgroup.json
07:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T298555)', diff saved to https://phabricator.wikimedia.org/P28413 and previous config saved to /var/cache/conftool/dbconfig/20220524-072243-ladsgroup.json
07:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P28412 and previous config saved to /var/cache/conftool/dbconfig/20220524-071557-ladsgroup.json
07:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298555)', diff saved to https://phabricator.wikimedia.org/P28411 and previous config saved to /var/cache/conftool/dbconfig/20220524-070052-ladsgroup.json
06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28410 and previous config saved to /var/cache/conftool/dbconfig/20220524-065643-ladsgroup.json
06:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
06:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance
06:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298560)', diff saved to https://phabricator.wikimedia.org/P28409 and previous config saved to /var/cache/conftool/dbconfig/20220524-065635-ladsgroup.json
06:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P28408 and previous config saved to /var/cache/conftool/dbconfig/20220524-064130-ladsgroup.json
06:26 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P28407 and previous config saved to /var/cache/conftool/dbconfig/20220524-062625-ladsgroup.json
06:25 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28406 and previous config saved to /var/cache/conftool/dbconfig/20220524-062531-root.json
06:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1157 (T298555)', diff saved to https://phabricator.wikimedia.org/P28405 and previous config saved to /var/cache/conftool/dbconfig/20220524-061237-ladsgroup.json
06:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1157.eqiad.wmnet with reason: Maintenance
06:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1157.eqiad.wmnet with reason: Maintenance
06:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1110 (T298560)', diff saved to https://phabricator.wikimedia.org/P28404 and previous config saved to /var/cache/conftool/dbconfig/20220524-061119-ladsgroup.json
06:10 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 75%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28403 and previous config saved to /var/cache/conftool/dbconfig/20220524-061027-root.json
06:08 marostegui: Rename revision_actor_temp on s8 T307906
05:55 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 50%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28402 and previous config saved to /var/cache/conftool/dbconfig/20220524-055523-root.json
05:40 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 25%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28401 and previous config saved to /var/cache/conftool/dbconfig/20220524-054019-root.json
05:25 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 10%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28400 and previous config saved to /var/cache/conftool/dbconfig/20220524-052515-root.json
05:11 marostegui: Rename revision_actor_temp on s6 T307906
05:10 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 5%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28399 and previous config saved to /var/cache/conftool/dbconfig/20220524-051011-root.json
04:56 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T298555)', diff saved to https://phabricator.wikimedia.org/P28398 and previous config saved to /var/cache/conftool/dbconfig/20220524-045602-ladsgroup.json
04:56 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
04:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
04:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1156.eqiad.wmnet with reason: Maintenance
04:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1156.eqiad.wmnet with reason: Maintenance
04:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298555)', diff saved to https://phabricator.wikimedia.org/P28397 and previous config saved to /var/cache/conftool/dbconfig/20220524-045549-ladsgroup.json
04:55 marostegui@cumin1001: dbctl commit (dc=all): 'db1172 (re)pooling @ 1%: After migrating back to 10.4', diff saved to https://phabricator.wikimedia.org/P28396 and previous config saved to /var/cache/conftool/dbconfig/20220524-045508-root.json
04:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P28395 and previous config saved to /var/cache/conftool/dbconfig/20220524-044044-ladsgroup.json
04:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P28394 and previous config saved to /var/cache/conftool/dbconfig/20220524-042539-ladsgroup.json
04:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122 (T298555)', diff saved to https://phabricator.wikimedia.org/P28393 and previous config saved to /var/cache/conftool/dbconfig/20220524-041034-ladsgroup.json
03:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
03:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
03:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298555)', diff saved to https://phabricator.wikimedia.org/P28392 and previous config saved to /var/cache/conftool/dbconfig/20220524-032848-ladsgroup.json
03:13 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P28391 and previous config saved to /var/cache/conftool/dbconfig/20220524-031343-ladsgroup.json
02:58 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P28390 and previous config saved to /var/cache/conftool/dbconfig/20220524-025838-ladsgroup.json
02:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298555)', diff saved to https://phabricator.wikimedia.org/P28389 and previous config saved to /var/cache/conftool/dbconfig/20220524-024333-ladsgroup.json
02:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
02:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
02:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
02:32 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
02:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
02:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
02:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
01:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T298555)', diff saved to https://phabricator.wikimedia.org/P28388 and previous config saved to /var/cache/conftool/dbconfig/20220524-015145-ladsgroup.json
01:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1122.eqiad.wmnet with reason: Maintenance
01:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1122.eqiad.wmnet with reason: Maintenance
01:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298555)', diff saved to https://phabricator.wikimedia.org/P28387 and previous config saved to /var/cache/conftool/dbconfig/20220524-015137-ladsgroup.json
01:37 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host relforge1004.eqiad.wmnet with OS bullseye
01:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P28386 and previous config saved to /var/cache/conftool/dbconfig/20220524-013632-ladsgroup.json
01:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P28385 and previous config saved to /var/cache/conftool/dbconfig/20220524-012127-ladsgroup.json
01:19 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1004.eqiad.wmnet with reason: host reimage
01:16 ryankemper@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1004.eqiad.wmnet with reason: host reimage
01:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1110 (T298560)', diff saved to https://phabricator.wikimedia.org/P28384 and previous config saved to /var/cache/conftool/dbconfig/20220524-010810-ladsgroup.json
01:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
01:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1110.eqiad.wmnet with reason: Maintenance
01:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28383 and previous config saved to /var/cache/conftool/dbconfig/20220524-010802-ladsgroup.json
01:06 ryankemper@cumin1001: START - Cookbook sre.hosts.reimage for host relforge1004.eqiad.wmnet with OS bullseye
01:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298555)', diff saved to https://phabricator.wikimedia.org/P28382 and previous config saved to /var/cache/conftool/dbconfig/20220524-010622-ladsgroup.json
01:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1112 (T298555)', diff saved to https://phabricator.wikimedia.org/P28381 and previous config saved to /var/cache/conftool/dbconfig/20220524-010534-ladsgroup.json
01:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
01:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
01:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1112.eqiad.wmnet with reason: Maintenance
01:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1112.eqiad.wmnet with reason: Maintenance
01:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298555)', diff saved to https://phabricator.wikimedia.org/P28380 and previous config saved to /var/cache/conftool/dbconfig/20220524-010521-ladsgroup.json
00:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P28379 and previous config saved to /var/cache/conftool/dbconfig/20220524-005257-ladsgroup.json
00:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P28378 and previous config saved to /var/cache/conftool/dbconfig/20220524-005016-ladsgroup.json
00:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P28377 and previous config saved to /var/cache/conftool/dbconfig/20220524-003752-ladsgroup.json
00:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P28376 and previous config saved to /var/cache/conftool/dbconfig/20220524-003511-ladsgroup.json
00:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28375 and previous config saved to /var/cache/conftool/dbconfig/20220524-002246-ladsgroup.json
00:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T298555)', diff saved to https://phabricator.wikimedia.org/P28374 and previous config saved to /var/cache/conftool/dbconfig/20220524-002006-ladsgroup.json

2022-05-23

23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T298555)', diff saved to https://phabricator.wikimedia.org/P28373 and previous config saved to /var/cache/conftool/dbconfig/20220523-235415-ladsgroup.json
23:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1129.eqiad.wmnet with reason: Maintenance
23:54 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1129.eqiad.wmnet with reason: Maintenance
23:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298555)', diff saved to https://phabricator.wikimedia.org/P28372 and previous config saved to /var/cache/conftool/dbconfig/20220523-235407-ladsgroup.json
23:49 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@02f2375]: increase driver jvm heap for convert_to_esbulk (duration: 02m 18s)
23:47 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@02f2375]: increase driver jvm heap for convert_to_esbulk
23:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P28371 and previous config saved to /var/cache/conftool/dbconfig/20220523-233902-ladsgroup.json
23:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P28370 and previous config saved to /var/cache/conftool/dbconfig/20220523-232357-ladsgroup.json
23:20 mutante: cumin1001 - systemtl start httpbb_hourly_appserver after deploying gerrit:797533 leads to '+icinga-wm> RECOVERY - Check systemd state on cumin1001 is OK: OK" T116948
23:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298555)', diff saved to https://phabricator.wikimedia.org/P28369 and previous config saved to /var/cache/conftool/dbconfig/20220523-230851-ladsgroup.json
22:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1179 (T298555)', diff saved to https://phabricator.wikimedia.org/P28368 and previous config saved to /var/cache/conftool/dbconfig/20220523-224119-ladsgroup.json
22:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1179.eqiad.wmnet with reason: Maintenance
22:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1179.eqiad.wmnet with reason: Maintenance
22:12 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host relforge1003.eqiad.wmnet with OS bullseye
21:54 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on relforge1003.eqiad.wmnet with reason: host reimage
21:50 bking@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on relforge1003.eqiad.wmnet with reason: host reimage
21:43 mutante: [cumin1001:~] $ sudo systemctl start httpbb_hourly_appserver
21:40 bking@cumin1001: START - Cookbook sre.hosts.reimage for host relforge1003.eqiad.wmnet with OS bullseye
21:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
21:04 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
21:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1182 (T298555)', diff saved to https://phabricator.wikimedia.org/P28367 and previous config saved to /var/cache/conftool/dbconfig/20220523-210339-ladsgroup.json
21:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1182.eqiad.wmnet with reason: Maintenance
21:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1182.eqiad.wmnet with reason: Maintenance
21:00 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:49 cjming: end of UTC late backport window
20:48 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:48 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@2f7ddb1]: increase driver memory_overhead for convert_to_esbulk (duration: 02m 20s)
20:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:47 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Deploy TOC A/B test to frwiki, ptwiki at 50% (T306607) (duration: 00m 52s)
20:46 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@2f7ddb1]: increase driver memory_overhead for convert_to_esbulk
20:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:41 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:41 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:40 jforrester@deploy1002: Synchronized wmf-config/extension-list: Config: Drop CodeReview, Part III: Drop from i18n build step (T116948) (duration: 00m 51s)
20:37 jforrester@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Drop CodeReview, Part II: Stop configuring it anywhere (T116948) (duration: 00m 51s)
20:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:34 jforrester@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Drop CodeReview, Part I: Stop loading it anywhere (T116948) (duration: 00m 51s)
20:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:28 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Add localized wordmark for plwiktionary (T307683) (duration: 00m 51s)
20:28 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:27 cjming@deploy1002: Synchronized static/images/mobile/copyright/wiktionary-wordmark-pl.svg: Config: Add localized wordmark for plwiktionary (T307683) (duration: 00m 50s)
20:24 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@aa49833]: increase memory_overhead for convert_to_esbulk (duration: 02m 24s)
20:23 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:22 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@aa49833]: increase memory_overhead for convert_to_esbulk
20:21 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Start writing to cuc_actor in test wikis (T233004) (duration: 00m 50s)
20:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:13 cjming@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: commonswiki: Enable wgCopyUploadAllowOnWikiDomainConfig (T300407) (duration: 00m 52s)
20:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 8 hosts with reason: Maintenance
20:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 8 hosts with reason: Maintenance
20:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2104.codfw.wmnet with reason: Maintenance
20:10 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2104.codfw.wmnet with reason: Maintenance
20:10 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
19:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1113:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28366 and previous config saved to /var/cache/conftool/dbconfig/20220523-194659-ladsgroup.json
19:46 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
19:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1113.eqiad.wmnet with reason: Maintenance
19:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 6 hosts with reason: Maintenance
19:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 6 hosts with reason: Maintenance
19:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2105.codfw.wmnet with reason: Maintenance
19:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2105.codfw.wmnet with reason: Maintenance
19:29 kharlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
19:26 kharlan@deploy1002: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
19:26 kharlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
19:23 kharlan@deploy1002: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
19:21 kharlan@deploy1002: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
19:19 kharlan@deploy1002: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
19:19 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@5a4803a]: T307983: zero-pad dates within @dailysnapshot (duration: 02m 20s)
19:17 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@5a4803a]: T307983: zero-pad dates within @dailysnapshot
18:32 mforns@deploy1002: Finished deploy [airflow-dags/analytics@2d8e8d1]: (no justification provided) (duration: 00m 07s)
18:32 mforns@deploy1002: Started deploy [airflow-dags/analytics@2d8e8d1]: (no justification provided)
18:25 ryankemper: T308647 Bringing `elastic2054` back into service: `ryankemper@elastic2054:~$ sudo pool` (it's not currently banned from cluster so nothing to do there)
18:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T303171)', diff saved to https://phabricator.wikimedia.org/P28364 and previous config saved to /var/cache/conftool/dbconfig/20220523-181954-ladsgroup.json
18:08 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@d1f4367]: T307983: weekly import of image suggestions (duration: 02m 21s)
18:07 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1139.eqiad.wmnet with reason: Maintenance
18:07 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1139.eqiad.wmnet with reason: Maintenance
18:05 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@d1f4367]: T307983: weekly import of image suggestions
18:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P28361 and previous config saved to /var/cache/conftool/dbconfig/20220523-180449-ladsgroup.json
17:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P28360 and previous config saved to /var/cache/conftool/dbconfig/20220523-174944-ladsgroup.json
17:36 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106 (T303171)', diff saved to https://phabricator.wikimedia.org/P28359 and previous config saved to /var/cache/conftool/dbconfig/20220523-173439-ladsgroup.json
17:30 pt1979@cumin2002: START - Cookbook sre.dns.netbox
17:26 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1106.eqiad.wmnet with OS bullseye
17:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1106.eqiad.wmnet with reason: host reimage
17:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1106.eqiad.wmnet with reason: host reimage
17:04 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:00 bking@cumin1001: START - Cookbook sre.dns.netbox
16:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1106.eqiad.wmnet with OS bullseye
16:59 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:52 pt1979@cumin2002: START - Cookbook sre.dns.netbox
16:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T303171)', diff saved to https://phabricator.wikimedia.org/P28358 and previous config saved to /var/cache/conftool/dbconfig/20220523-165045-ladsgroup.json
16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
16:50 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance
16:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T303171)', diff saved to https://phabricator.wikimedia.org/P28357 and previous config saved to /var/cache/conftool/dbconfig/20220523-164621-ladsgroup.json
16:44 inflatador: add AAAA records to elastic202[5-9] T271143
16:39 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti5001.eqsin.wmnet with OS bullseye
16:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P28356 and previous config saved to /var/cache/conftool/dbconfig/20220523-163116-ladsgroup.json
16:29 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
16:23 pt1979@cumin2002: START - Cookbook sre.dns.netbox
16:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2012.codfw.wmnet
16:17 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5001.eqsin.wmnet with reason: host reimage
16:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2012.codfw.wmnet
16:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P28355 and previous config saved to /var/cache/conftool/dbconfig/20220523-161610-ladsgroup.json
16:13 robh@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5001.eqsin.wmnet with reason: host reimage
16:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1102.eqiad.wmnet with reason: Maintenance
16:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1102.eqiad.wmnet with reason: Maintenance
16:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298555)', diff saved to https://phabricator.wikimedia.org/P28354 and previous config saved to /var/cache/conftool/dbconfig/20220523-160341-ladsgroup.json
16:01 cmooney@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs1020.eqiad.wmnet with OS bullseye
16:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1102.eqiad.wmnet with reason: Maintenance
16:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1102.eqiad.wmnet with reason: Maintenance
16:01 volans@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS buster
16:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T303171)', diff saved to https://phabricator.wikimedia.org/P28353 and previous config saved to /var/cache/conftool/dbconfig/20220523-160105-ladsgroup.json
15:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2011.codfw.wmnet
15:53 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2011.codfw.wmnet
15:49 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1184.eqiad.wmnet with OS bullseye
15:48 volans@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage
15:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P28352 and previous config saved to /var/cache/conftool/dbconfig/20220523-154836-ladsgroup.json
15:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2010.codfw.wmnet
15:46 volans@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage
15:45 mbsantos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
15:44 mbsantos@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
15:44 mbsantos@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
15:43 robh@cumin1001: START - Cookbook sre.hosts.reimage for host ganeti5001.eqsin.wmnet with OS bullseye
15:43 mbsantos@deploy1002: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
15:43 mbsantos@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
15:42 mbsantos@deploy1002: helmfile [staging] START helmfile.d/services/wikifeeds: apply
15:42 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2010.codfw.wmnet
15:39 vgutierrez: pool cp2038 - T308459
15:38 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2038.codfw.wmnet
15:38 vgutierrez@cumin1001: START - Cookbook sre.hosts.remove-downtime for cp2038.codfw.wmnet
15:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
15:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
15:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
15:35 volans@cumin1001: START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS buster
15:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
15:34 volans@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host sretest1001.eqiad.wmnet with OS buster
15:34 volans@cumin1001: START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS buster
15:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P28351 and previous config saved to /var/cache/conftool/dbconfig/20220523-153331-ladsgroup.json
15:32 cmooney@cumin1001: START - Cookbook sre.hosts.reimage for host aqs1020.eqiad.wmnet with OS bullseye
15:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2009.codfw.wmnet
15:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage
15:30 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
15:30 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
15:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1184.eqiad.wmnet with reason: host reimage
15:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2009.codfw.wmnet
15:26 taavi: deploy patch for T309028
15:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T298555)', diff saved to https://phabricator.wikimedia.org/P28350 and previous config saved to /var/cache/conftool/dbconfig/20220523-151826-ladsgroup.json
15:17 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 100%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28349 and previous config saved to /var/cache/conftool/dbconfig/20220523-151721-root.json
15:17 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1184.eqiad.wmnet with OS bullseye
15:14 jbond@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netbox1002.eqiad.wmnet
15:13 papaul: poweroff cp2038 for maintenance
15:12 robh: updating firmware on ganeti5001 per T308211
15:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1184 (T303171)', diff saved to https://phabricator.wikimedia.org/P28348 and previous config saved to /var/cache/conftool/dbconfig/20220523-151207-ladsgroup.json
15:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance
15:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance
15:07 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T303171)', diff saved to https://phabricator.wikimedia.org/P28346 and previous config saved to /var/cache/conftool/dbconfig/20220523-150717-ladsgroup.json
15:06 jbond@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:02 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 75%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28345 and previous config saved to /var/cache/conftool/dbconfig/20220523-150217-root.json
15:02 Emperor: rebooting ms-be2069 to look at disk config
15:01 jbond@cumin1001: START - Cookbook sre.dns.netbox
15:01 jbond@cumin1001: START - Cookbook sre.ganeti.makevm for new host netbox1002.eqiad.wmnet
14:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P28343 and previous config saved to /var/cache/conftool/dbconfig/20220523-145212-ladsgroup.json
14:49 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti1024.eqiad.wmnet
14:47 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 50%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28342 and previous config saved to /var/cache/conftool/dbconfig/20220523-144713-root.json
14:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1024.eqiad.wmnet
14:37 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P28341 and previous config saved to /var/cache/conftool/dbconfig/20220523-143707-ladsgroup.json
14:36 hnowlan@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host maps2009.codfw.wmnet
14:34 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:32 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 25%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28340 and previous config saved to /var/cache/conftool/dbconfig/20220523-143209-root.json
14:30 hnowlan@cumin1001: START - Cookbook sre.hosts.reboot-single for host maps2009.codfw.wmnet
14:26 bking@cumin1001: START - Cookbook sre.dns.netbox
14:26 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2010.codfw.wmnet
14:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135 (T303171)', diff saved to https://phabricator.wikimedia.org/P28339 and previous config saved to /var/cache/conftool/dbconfig/20220523-142202-ladsgroup.json
14:20 inflatador: Add AAAA records to relforge1003 and 1004 T271143
14:20 hnowlan@cumin1001: START - Cookbook sre.hosts.reboot-single for host maps2010.codfw.wmnet
14:18 moritzm: failover ganeti master in eqiad to ganeti1027
14:17 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 10%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28338 and previous config saved to /var/cache/conftool/dbconfig/20220523-141705-root.json
14:14 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1135.eqiad.wmnet with OS bullseye
14:12 aqu@deploy1002: Finished deploy [airflow-dags/analytics@95d0f86]: T295072 spark 3 from airflow venv pyspark [airflow-dags/analytics@95d0f86] (duration: 00m 08s)
14:12 aqu@deploy1002: Started deploy [airflow-dags/analytics@95d0f86]: T295072 spark 3 from airflow venv pyspark [airflow-dags/analytics@95d0f86]
14:10 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1175 (T298555)', diff saved to https://phabricator.wikimedia.org/P28337 and previous config saved to /var/cache/conftool/dbconfig/20220523-141001-ladsgroup.json
14:10 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1175.eqiad.wmnet with reason: Maintenance
14:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298555)', diff saved to https://phabricator.wikimedia.org/P28336 and previous config saved to /var/cache/conftool/dbconfig/20220523-140954-ladsgroup.json
14:08 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@95d0f86]: T295072 Spark 3 from Airflow venv pyspark [airflow-dags/analytics_test@95d0f86] (duration: 00m 08s)
14:08 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@95d0f86]: T295072 Spark 3 from Airflow venv pyspark [airflow-dags/analytics_test@95d0f86]
14:02 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2008.codfw.wmnet
14:02 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 5%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28335 and previous config saved to /var/cache/conftool/dbconfig/20220523-140201-root.json
14:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
14:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
14:01 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28334 and previous config saved to /var/cache/conftool/dbconfig/20220523-140156-ladsgroup.json
14:00 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet
13:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1135.eqiad.wmnet with reason: host reimage
13:57 hnowlan@cumin1001: START - Cookbook sre.hosts.reboot-single for host maps2008.codfw.wmnet
13:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
13:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1135.eqiad.wmnet with reason: host reimage
13:55 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2007.codfw.wmnet
13:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P28332 and previous config saved to /var/cache/conftool/dbconfig/20220523-135449-ladsgroup.json
13:49 hnowlan@cumin1001: START - Cookbook sre.hosts.reboot-single for host maps2007.codfw.wmnet
13:46 marostegui@cumin1001: dbctl commit (dc=all): 'db1177 (re)pooling @ 1%: After recloning db1172', diff saved to https://phabricator.wikimedia.org/P28331 and previous config saved to /var/cache/conftool/dbconfig/20220523-134657-root.json
13:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P28330 and previous config saved to /var/cache/conftool/dbconfig/20220523-134651-ladsgroup.json
13:46 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1135.eqiad.wmnet with OS bullseye
13:46 tgr: EU mid-day deploys done
13:45 tgr@deploy1002: Synchronized php-1.39.0-wmf.12/extensions/OAuth/src/Frontend/SpecialPages/SpecialMWOAuthConsumerRegistration.php: Backport: Remove 'required' from callbackIsPrefix (T308880) (duration: 00m 50s)
13:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:41 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:41 tgr@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: rowiki: Use Romanian canonical name (T127607) (duration: 00m 50s)
13:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P28329 and previous config saved to /var/cache/conftool/dbconfig/20220523-133944-ladsgroup.json
13:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:35 tgr@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: itwiki: Add "editautopatrolprotected" protection level (T308917) (duration: 00m 52s)
13:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1135 (T303171)', diff saved to https://phabricator.wikimedia.org/P28328 and previous config saved to /var/cache/conftool/dbconfig/20220523-133228-ladsgroup.json
13:32 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1135.eqiad.wmnet with reason: Maintenance
13:32 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1135.eqiad.wmnet with reason: Maintenance
13:31 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P28327 and previous config saved to /var/cache/conftool/dbconfig/20220523-133146-ladsgroup.json
13:30 tgr@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: zhwiki: Enable RCPatrol (T308976) (duration: 00m 51s)
13:25 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T303171)', diff saved to https://phabricator.wikimedia.org/P28326 and previous config saved to /var/cache/conftool/dbconfig/20220523-132459-ladsgroup.json
13:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T298555)', diff saved to https://phabricator.wikimedia.org/P28325 and previous config saved to /var/cache/conftool/dbconfig/20220523-132438-ladsgroup.json
13:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:16 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28324 and previous config saved to /var/cache/conftool/dbconfig/20220523-131641-ladsgroup.json
13:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:15 tgr@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Update IP addresses for Wiki Education Dashboard exemptions (T308702) (duration: 00m 52s)
13:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P28323 and previous config saved to /var/cache/conftool/dbconfig/20220523-130954-ladsgroup.json
12:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P28322 and previous config saved to /var/cache/conftool/dbconfig/20220523-125449-ladsgroup.json
12:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti5001.eqsin.wmnet with reason: Remove from cluster for firmware update and eventual reimage
12:51 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti5001.eqsin.wmnet with reason: Remove from cluster for firmware update and eventual reimage
12:39 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1134 (T303171)', diff saved to https://phabricator.wikimedia.org/P28321 and previous config saved to /var/cache/conftool/dbconfig/20220523-123944-ladsgroup.json
12:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet
12:25 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1134.eqiad.wmnet with OS bullseye
12:23 aqu@deploy1002: Finished deploy [airflow-dags/analytics@c9b397c]: T305843_migrate_clickstream_job_from_oozie_to_airflow [airflow-dags/analytics@c9b397c] (duration: 00m 08s)
12:23 aqu@deploy1002: Started deploy [airflow-dags/analytics@c9b397c]: T305843_migrate_clickstream_job_from_oozie_to_airflow [airflow-dags/analytics@c9b397c]
12:20 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@c9b397c]: T305843_migrate_clickstream_job_from_oozie_to_airflow [airflow-dags/analytics_test@c9b397c] (duration: 00m 08s)
12:20 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@c9b397c]: T305843_migrate_clickstream_job_from_oozie_to_airflow [airflow-dags/analytics_test@c9b397c]
12:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1166 (T298555)', diff saved to https://phabricator.wikimedia.org/P28320 and previous config saved to /var/cache/conftool/dbconfig/20220523-121659-ladsgroup.json
12:16 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1166.eqiad.wmnet with reason: Maintenance
12:16 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1166.eqiad.wmnet with reason: Maintenance
12:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1134.eqiad.wmnet with reason: host reimage
12:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1134.eqiad.wmnet with reason: host reimage
12:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet
12:01 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2006.codfw.wmnet
11:56 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1134.eqiad.wmnet with OS bullseye
11:56 hnowlan@cumin1001: START - Cookbook sre.hosts.reboot-single for host maps2006.codfw.wmnet
11:52 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2005.codfw.wmnet
11:52 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1134 (T303171)', diff saved to https://phabricator.wikimedia.org/P28318 and previous config saved to /var/cache/conftool/dbconfig/20220523-115202-ladsgroup.json
11:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance
11:51 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1134.eqiad.wmnet with reason: Maintenance
11:47 hnowlan@cumin1001: START - Cookbook sre.hosts.reboot-single for host maps2005.codfw.wmnet
11:46 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T303171)', diff saved to https://phabricator.wikimedia.org/P28317 and previous config saved to /var/cache/conftool/dbconfig/20220523-114559-ladsgroup.json
11:41 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti1026.eqiad.wmnet
11:38 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps1010.eqiad.wmnet
11:38 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on maps1010.eqiad.wmnet with reason: security update
11:38 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on maps1010.eqiad.wmnet with reason: security update
11:30 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P28316 and previous config saved to /var/cache/conftool/dbconfig/20220523-113053-ladsgroup.json
11:25 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on maps1009.eqiad.wmnet with reason: security update
11:25 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on maps1009.eqiad.wmnet with reason: security update
11:25 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps1008.eqiad.wmnet
11:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1177 to clone db1172', diff saved to https://phabricator.wikimedia.org/P28314 and previous config saved to /var/cache/conftool/dbconfig/20220523-111902-marostegui.json
11:18 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on maps1008.eqiad.wmnet with reason: security update
11:18 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on maps1008.eqiad.wmnet with reason: security update
11:15 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P28313 and previous config saved to /var/cache/conftool/dbconfig/20220523-111548-ladsgroup.json
11:11 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on maps1008.eqiad.wmnet with reason: security update
11:11 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on maps1008.eqiad.wmnet with reason: security update
11:11 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps1008.eqiad.wmnet
11:10 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps1007.eqiad.wmnet
11:01 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1026.eqiad.wmnet
11:00 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1119 (T303171)', diff saved to https://phabricator.wikimedia.org/P28312 and previous config saved to /var/cache/conftool/dbconfig/20220523-110043-ladsgroup.json
10:58 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti1025.eqiad.wmnet
10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28311 and previous config saved to /var/cache/conftool/dbconfig/20220523-105332-ladsgroup.json
10:53 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1105.eqiad.wmnet with reason: Maintenance
10:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1105.eqiad.wmnet with reason: Maintenance
10:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28310 and previous config saved to /var/cache/conftool/dbconfig/20220523-105324-ladsgroup.json
10:51 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1119.eqiad.wmnet with OS bullseye
10:40 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on maps1007.eqiad.wmnet with reason: security update
10:40 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on maps1007.eqiad.wmnet with reason: security update
10:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P28309 and previous config saved to /var/cache/conftool/dbconfig/20220523-103819-ladsgroup.json
10:37 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps1006.eqiad.wmnet
10:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1119.eqiad.wmnet with reason: host reimage
10:33 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1119.eqiad.wmnet with reason: host reimage
10:25 btullis@deploy1002: Finished deploy [analytics/superset/deploy@09094de]: (no justification provided) (duration: 00m 03s)
10:25 btullis@deploy1002: Started deploy [analytics/superset/deploy@09094de]: (no justification provided)
10:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db1119.eqiad.wmnet with OS bullseye
10:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P28308 and previous config saved to /var/cache/conftool/dbconfig/20220523-102314-ladsgroup.json
10:18 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1025.eqiad.wmnet
10:18 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps1006.eqiad.wmnet
10:17 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on maps1006.eqiad.wmnet with reason: security update
10:17 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on maps1006.eqiad.wmnet with reason: security update
10:17 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps1005.eqiad.wmnet
10:12 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1119 (T303171)', diff saved to https://phabricator.wikimedia.org/P28307 and previous config saved to /var/cache/conftool/dbconfig/20220523-101222-ladsgroup.json
10:12 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1119.eqiad.wmnet with reason: Maintenance
10:12 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1119.eqiad.wmnet with reason: Maintenance
10:10 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 6 hosts with reason: postgres config change
10:10 hnowlan@cumin1001: START - Cookbook sre.hosts.downtime for 0:15:00 on 6 hosts with reason: postgres config change
10:09 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps1005.eqiad.wmnet
10:09 hnowlan: starting reboot of eqiad maps hosts for updates
10:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28306 and previous config saved to /var/cache/conftool/dbconfig/20220523-100809-ladsgroup.json
10:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet
10:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet
10:00 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab1003.wikimedia.org with OS bullseye
09:55 moritzm: drain ganeti5001 T308211
09:54 moritzm: failover ganeti master in eqsin to ganeti5003 T308211
09:49 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:44 volans@cumin1001: START - Cookbook sre.dns.netbox
09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1022.eqiad.wmnet
09:40 jelto@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1003.wikimedia.org with reason: host reimage
09:38 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1022.eqiad.wmnet
09:37 jelto@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1003.wikimedia.org with reason: host reimage
09:25 jelto@cumin1001: START - Cookbook sre.hosts.reimage for host gitlab1003.wikimedia.org with OS bullseye
09:25 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti5003.eqsin.wmnet to ganeti01.svc.eqsin.wmnet
09:24 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti5003.eqsin.wmnet to ganeti01.svc.eqsin.wmnet
09:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5003.eqsin.wmnet
09:13 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti5003.eqsin.wmnet
09:01 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1145.eqiad.wmnet with reason: Maintenance
09:01 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1145.eqiad.wmnet with reason: Maintenance
08:12 taavi: fixing renames of 44 accounts T308895
08:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
08:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
08:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
08:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
08:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298555)', diff saved to https://phabricator.wikimedia.org/P28303 and previous config saved to /var/cache/conftool/dbconfig/20220523-080244-ladsgroup.json
07:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:55 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:55 urbanecm@deploy1002: Synchronized docroot/noc/conf/activeMWVersions.php: e1df8fabc: phpcs: move ForbiddenFunctions.exec exclusion inline (duration: 00m 50s)
07:53 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 86d08457: phpcs: move ForbiddenFunctions.extract exclusion inline (duration: 00m 50s)
07:53 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host htmldumper1001.eqiad.wmnet
07:52 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: a888904: phpcs: enable and suppress ClassMatchesFilename.NotMatch (duration: 00m 49s)
07:51 urbanecm@deploy1002: Synchronized w/fatal-error.php: a888904: phpcs: enable and suppress ClassMatchesFilename.NotMatch (duration: 00m 49s)
07:50 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:50 urbanecm@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: 0e012139: phpcs: enable PropertyDocumentation.MissingDocumentationPrivate (duration: 00m 50s)
07:49 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host htmldumper1001.eqiad.wmnet
07:49 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:49 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:49 urbanecm@deploy1002: Synchronized w/fatal-error.php: 0e012139: phpcs: enable PropertyDocumentation.MissingDocumentationPrivate (duration: 00m 49s)
07:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28302 and previous config saved to /var/cache/conftool/dbconfig/20220523-074837-ladsgroup.json
07:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1146.eqiad.wmnet with reason: Maintenance
07:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1146.eqiad.wmnet with reason: Maintenance
07:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28301 and previous config saved to /var/cache/conftool/dbconfig/20220523-074829-ladsgroup.json
07:48 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:48 urbanecm@deploy1002: Synchronized src/: 0e012139: phpcs: enable PropertyDocumentation.MissingDocumentationPrivate (duration: 00m 50s)
07:47 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P28300 and previous config saved to /var/cache/conftool/dbconfig/20220523-074739-ladsgroup.json
07:43 urbanecm@deploy1002: Synchronized w/fatal-error.php: 7c28808: phpcs: enable and suppress DuplicateClassName.Found (duration: 00m 48s)
07:43 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:42 urbanecm@deploy1002: Synchronized w/fatal-error.php: 8f8b04e0: phpcs: enable PropertyDocumentation.WrongStyle (duration: 00m 50s)
07:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:41 urbanecm@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: 8f8b04e0: phpcs: enable PropertyDocumentation.WrongStyle (duration: 00m 49s)
07:41 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:40 urbanecm@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: e6fb9266: phpcs: enable FunctionComment.MissingDocumentationPrivate (duration: 01m 30s)
07:38 urbanecm@deploy1002: Synchronized private/readme.php: 7a8d8a06: phpcs: move DisallowYodaConditions exclusion inline (duration: 00m 49s)
07:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P28299 and previous config saved to /var/cache/conftool/dbconfig/20220523-073324-ladsgroup.json
07:32 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P28298 and previous config saved to /var/cache/conftool/dbconfig/20220523-073233-ladsgroup.json
07:25 kartik@deploy1002: Synchronized php-1.39.0-wmf.12/extensions/ContentTranslation/modules/base/mw.cx.SiteMapper.js: Backport: Sitemapper: Fix the configuration override (T308802) (duration: 00m 51s)
07:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:23 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:23 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P28297 and previous config saved to /var/cache/conftool/dbconfig/20220523-071819-ladsgroup.json
07:17 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298555)', diff saved to https://phabricator.wikimedia.org/P28296 and previous config saved to /var/cache/conftool/dbconfig/20220523-071728-ladsgroup.json
07:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
07:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
07:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
07:13 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P28295 and previous config saved to /var/cache/conftool/dbconfig/20220523-071334-root.json
07:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
07:10 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable ContentTranslation as default for cs, el, he, ko and tr WPs (T298239 T304853 T304854 T304855 T304863) (duration: 00m 50s)
07:09 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Razzi out of all services on: 1227 hosts
07:09 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Razzi out of all services on: 1227 hosts
07:09 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Razzi out of all services on: 562 hosts
07:08 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Razzi out of all services on: 562 hosts
07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28294 and previous config saved to /var/cache/conftool/dbconfig/20220523-070314-ladsgroup.json
06:58 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
06:58 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance
06:58 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P28293 and previous config saved to /var/cache/conftool/dbconfig/20220523-065830-root.json
06:51 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
06:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
06:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
06:49 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
06:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
06:43 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P28292 and previous config saved to /var/cache/conftool/dbconfig/20220523-064326-root.json
06:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
06:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
06:39 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
06:38 urbanecm@deploy1002: Synchronized private/PrivateSettings.php: Update T250887 mitigations (duration: 00m 52s)
06:35 ladsgroup@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Remove unused OggThumbLocation config variable (T308191) (duration: 00m 51s)
06:34 urbanecm: urbanecm@mwmaint1002:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057
06:28 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P28291 and previous config saved to /var/cache/conftool/dbconfig/20220523-062822-root.json
06:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1172.eqiad.wmnet with OS bullseye
06:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
06:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
06:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
06:13 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P28290 and previous config saved to /var/cache/conftool/dbconfig/20220523-061319-root.json
06:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
06:10 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Turn on WRITE BOTH for templatelink migration in enwiki (T299421) (duration: 00m 51s)
06:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
06:07 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: TimedMediaHandler: Drop pre-switch config, no longer read (T248418) (duration: 00m 54s)
06:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
06:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
06:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage
06:04 ladsgroup@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: TimedMediaHandler: Don't read wmgTmhWebPlayer (T248418) (duration: 00m 50s)
06:03 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
06:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1172.eqiad.wmnet with reason: host reimage
06:02 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: TimedMediaHandler: Drop Beta Feature, no longer usable (T248418) (duration: 00m 52s)
06:00 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: TimedMediaHandler: Disabled the BetaFeature from wikis (T248418) (duration: 00m 51s)
05:58 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 5%: After reimage', diff saved to https://phabricator.wikimedia.org/P28289 and previous config saved to /var/cache/conftool/dbconfig/20220523-055815-root.json
05:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T303603)', diff saved to https://phabricator.wikimedia.org/P28288 and previous config saved to /var/cache/conftool/dbconfig/20220523-055140-ladsgroup.json
05:51 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1172.eqiad.wmnet with OS bullseye
05:43 marostegui@cumin1001: dbctl commit (dc=all): 'db1118 (re)pooling @ 1%: After reimage', diff saved to https://phabricator.wikimedia.org/P28287 and previous config saved to /var/cache/conftool/dbconfig/20220523-054311-root.json
05:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P28286 and previous config saved to /var/cache/conftool/dbconfig/20220523-053635-ladsgroup.json
05:35 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1118.eqiad.wmnet with OS bullseye
05:33 kart_: Updated cxserver to 2022-05-22-062659-production (T290847)
05:31 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
05:30 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
05:28 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
05:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
05:27 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply
05:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
05:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
05:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
05:25 taavi@deploy1002: Synchronized php-1.39.0-wmf.12/extensions/WikimediaMaintenance/fixT308895BrokenRenames.php: Backport: Add a script to fix T308895 renames (T308895) (duration: 00m 51s)
05:24 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
05:24 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply
05:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P28285 and previous config saved to /var/cache/conftool/dbconfig/20220523-052130-ladsgroup.json
05:18 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1118.eqiad.wmnet with reason: host reimage
05:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on db1118.eqiad.wmnet with reason: host reimage
05:06 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db1118.eqiad.wmnet with OS bullseye
05:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1131 (T303603)', diff saved to https://phabricator.wikimedia.org/P28284 and previous config saved to /var/cache/conftool/dbconfig/20220523-050624-ladsgroup.json
05:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1131 (T303603)', diff saved to https://phabricator.wikimedia.org/P28283 and previous config saved to /var/cache/conftool/dbconfig/20220523-050341-ladsgroup.json
05:03 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
05:03 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance
04:58 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1118 reimage to bulseye', diff saved to https://phabricator.wikimedia.org/P28282 and previous config saved to /var/cache/conftool/dbconfig/20220523-045850-marostegui.json
04:55 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T298555)', diff saved to https://phabricator.wikimedia.org/P28281 and previous config saved to /var/cache/conftool/dbconfig/20220523-045548-ladsgroup.json
04:55 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1170.eqiad.wmnet with reason: Maintenance
04:55 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1170.eqiad.wmnet with reason: Maintenance
04:54 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298555)', diff saved to https://phabricator.wikimedia.org/P28280 and previous config saved to /var/cache/conftool/dbconfig/20220523-045404-ladsgroup.json
04:54 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1143.eqiad.wmnet with reason: Maintenance
04:53 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1143.eqiad.wmnet with reason: Maintenance
04:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0)
04:40 marostegui@cumin1001: Updating IPMI password on 1 hosts - marostegui@cumin1001
04:40 marostegui@cumin1001: START - Cookbook sre.hosts.ipmi-password-reset
04:40 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99)
04:39 marostegui@cumin1001: START - Cookbook sre.hosts.ipmi-password-reset

2022-05-22

20:46 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
20:42 krinkle@deploy1002: Synchronized wmf-config/: I14c5a9aa39 (duration: 00m 50s)
20:41 krinkle@deploy1002: Synchronized src/Profiler.php: I14c5a9aa39 (duration: 00m 49s)
20:34 krinkle@deploy1002: Synchronized lib/: I3882be35572 (duration: 00m 50s)
20:32 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
20:32 krinkle@deploy1002: Synchronized wmf-config/profiler.php: I3882be35572 (duration: 00m 51s)
20:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
20:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
20:31 krinkle@deploy1002: Synchronized src/XhguiSaverPdo.php: I3882be35572 (duration: 00m 50s)
20:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
18:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28278 and previous config saved to /var/cache/conftool/dbconfig/20220522-185021-ladsgroup.json
18:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P28277 and previous config saved to /var/cache/conftool/dbconfig/20220522-183516-ladsgroup.json
18:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P28276 and previous config saved to /var/cache/conftool/dbconfig/20220522-182011-ladsgroup.json
18:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28275 and previous config saved to /var/cache/conftool/dbconfig/20220522-180506-ladsgroup.json
17:14 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298555)', diff saved to https://phabricator.wikimedia.org/P28274 and previous config saved to /var/cache/conftool/dbconfig/20220522-171444-ladsgroup.json
14:49 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298555)', diff saved to https://phabricator.wikimedia.org/P28273 and previous config saved to /var/cache/conftool/dbconfig/20220522-144855-ladsgroup.json
14:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1138.eqiad.wmnet with reason: Maintenance
14:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1138.eqiad.wmnet with reason: Maintenance
14:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28272 and previous config saved to /var/cache/conftool/dbconfig/20220522-144847-ladsgroup.json
14:27 krinkle@deploy1002: Synchronized src/: Ia0a6d4 (duration: 00m 50s)
14:23 krinkle@deploy1002: Synchronized docroot/noc/: Ia0a6d4 (duration: 00m 50s)
14:18 krinkle@deploy1002: Synchronized wmf-config/: Ia0a6d4 (2/2) (duration: 00m 42s)
14:15 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:14 krinkle@deploy1002: scap failed: average error rate on 3/8 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details)
14:14 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:14 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
14:11 krinkle@deploy1002: Synchronized multiversion/: Ia0a6d4 (1/2) (duration: 00m 50s)
14:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
14:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
14:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
14:02 krinkle@deploy1002: Synchronized wmf-config/InitialiseSettings.php: I31b1bf (duration: 00m 52s)
13:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:40 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:40 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:36 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:28 krinkle@deploy1002: Synchronized multiversion/: I375917 (duration: 00m 53s)
13:25 krinkle@deploy1002: Synchronized wmf-config/CommonSettings.php: I97878f8e6 (duration: 00m 50s)
13:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
13:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
13:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
13:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
13:18 krinkle@deploy1002: Scap failed!: 7/8 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back.
13:17 krinkle@deploy1002: scap failed: average error rate on 7/8 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details)
12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28270 and previous config saved to /var/cache/conftool/dbconfig/20220522-122410-ladsgroup.json
12:24 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance
12:24 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance
12:24 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298555)', diff saved to https://phabricator.wikimedia.org/P28269 and previous config saved to /var/cache/conftool/dbconfig/20220522-122402-ladsgroup.json
10:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T298555)', diff saved to https://phabricator.wikimedia.org/P28267 and previous config saved to /var/cache/conftool/dbconfig/20220522-100436-ladsgroup.json
10:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1149.eqiad.wmnet with reason: Maintenance
10:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1149.eqiad.wmnet with reason: Maintenance
10:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298555)', diff saved to https://phabricator.wikimedia.org/P28266 and previous config saved to /var/cache/conftool/dbconfig/20220522-100429-ladsgroup.json
09:53 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T303603)', diff saved to https://phabricator.wikimedia.org/P28265 and previous config saved to /var/cache/conftool/dbconfig/20220522-095327-ladsgroup.json
09:38 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P28264 and previous config saved to /var/cache/conftool/dbconfig/20220522-093822-ladsgroup.json
09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28263 and previous config saved to /var/cache/conftool/dbconfig/20220522-093619-ladsgroup.json
09:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db1170.eqiad.wmnet with reason: Maintenance
09:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db1170.eqiad.wmnet with reason: Maintenance
09:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28262 and previous config saved to /var/cache/conftool/dbconfig/20220522-093611-ladsgroup.json
09:23 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P28261 and previous config saved to /var/cache/conftool/dbconfig/20220522-092317-ladsgroup.json
09:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P28260 and previous config saved to /var/cache/conftool/dbconfig/20220522-092106-ladsgroup.json
09:08 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T303603)', diff saved to https://phabricator.wikimedia.org/P28259 and previous config saved to /var/cache/conftool/dbconfig/20220522-090811-ladsgroup.json
09:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P28258 and previous config saved to /var/cache/conftool/dbconfig/20220522-090601-ladsgroup.json
08:50 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28257 and previous config saved to /var/cache/conftool/dbconfig/20220522-085056-ladsgroup.json
08:40 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1167 (T303603)', diff saved to https://phabricator.wikimedia.org/P28256 and previous config saved to /var/cache/conftool/dbconfig/20220522-084036-ladsgroup.json
08:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
08:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
08:40 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance
08:40 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance
07:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T298555)', diff saved to https://phabricator.wikimedia.org/P28255 and previous config saved to /var/cache/conftool/dbconfig/20220522-074303-ladsgroup.json
07:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1148.eqiad.wmnet with reason: Maintenance
07:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1148.eqiad.wmnet with reason: Maintenance
07:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298555)', diff saved to https://phabricator.wikimedia.org/P28254 and previous config saved to /var/cache/conftool/dbconfig/20220522-074255-ladsgroup.json
06:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T298555)', diff saved to https://phabricator.wikimedia.org/P28253 and previous config saved to /var/cache/conftool/dbconfig/20220522-064240-ladsgroup.json
06:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1143.eqiad.wmnet with reason: Maintenance
06:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1143.eqiad.wmnet with reason: Maintenance
06:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298555)', diff saved to https://phabricator.wikimedia.org/P28252 and previous config saved to /var/cache/conftool/dbconfig/20220522-064232-ladsgroup.json
05:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1127', diff saved to https://phabricator.wikimedia.org/P28251 and previous config saved to /var/cache/conftool/dbconfig/20220522-053905-marostegui.json
04:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T298555)', diff saved to https://phabricator.wikimedia.org/P28250 and previous config saved to /var/cache/conftool/dbconfig/20220522-042249-ladsgroup.json
04:22 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1142.eqiad.wmnet with reason: Maintenance
04:22 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1142.eqiad.wmnet with reason: Maintenance
02:13 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1145.eqiad.wmnet with reason: Maintenance
02:13 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1145.eqiad.wmnet with reason: Maintenance
00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28249 and previous config saved to /var/cache/conftool/dbconfig/20220522-002120-ladsgroup.json
00:21 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db1146.eqiad.wmnet with reason: Maintenance
00:21 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db1146.eqiad.wmnet with reason: Maintenance
00:21 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28248 and previous config saved to /var/cache/conftool/dbconfig/20220522-002112-ladsgroup.json
00:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P28247 and previous config saved to /var/cache/conftool/dbconfig/20220522-000607-ladsgroup.json
00:02 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
00:02 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
00:02 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298555)', diff saved to https://phabricator.wikimedia.org/P28246 and previous config saved to /var/cache/conftool/dbconfig/20220522-000225-ladsgroup.json

2022-05-21

23:51 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P28245 and previous config saved to /var/cache/conftool/dbconfig/20220521-235102-ladsgroup.json
23:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28244 and previous config saved to /var/cache/conftool/dbconfig/20220521-233556-ladsgroup.json
22:10 hashar: Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit # T308943
21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T298555)', diff saved to https://phabricator.wikimedia.org/P28243 and previous config saved to /var/cache/conftool/dbconfig/20220521-214346-ladsgroup.json
21:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1141.eqiad.wmnet with reason: Maintenance
21:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1141.eqiad.wmnet with reason: Maintenance
21:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298555)', diff saved to https://phabricator.wikimedia.org/P28242 and previous config saved to /var/cache/conftool/dbconfig/20220521-214338-ladsgroup.json
19:04 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298555)', diff saved to https://phabricator.wikimedia.org/P28241 and previous config saved to /var/cache/conftool/dbconfig/20220521-190446-ladsgroup.json
19:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
19:04 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1121.eqiad.wmnet with reason: Maintenance
19:04 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1121.eqiad.wmnet with reason: Maintenance
18:06 taavi: set rq_wiki = null for 26 rows in centralauth.renameuser_queue status table T308895
18:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
18:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
18:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:58 taavi@deploy1002: Synchronized php-1.39.0-wmf.12/extensions/CentralAuth: Backport: Revert "Populate rq_wiki with the wiki where the rename was requested" (T308895) (duration: 00m 51s)
17:43 krinkle@deploy1002: Synchronized multiversion/: I97878f (duration: 00m 51s)
17:39 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
17:38 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
17:38 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
17:37 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
17:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T303603)', diff saved to https://phabricator.wikimedia.org/P28240 and previous config saved to /var/cache/conftool/dbconfig/20220521-170638-ladsgroup.json
16:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on 12 hosts with reason: Maintenance
16:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 20:00:00 on 12 hosts with reason: Maintenance
16:48 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2110.codfw.wmnet with reason: Maintenance
16:48 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db2110.codfw.wmnet with reason: Maintenance
16:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28239 and previous config saved to /var/cache/conftool/dbconfig/20220521-164805-ladsgroup.json
16:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1178 (T303603)', diff saved to https://phabricator.wikimedia.org/P28238 and previous config saved to /var/cache/conftool/dbconfig/20220521-163639-ladsgroup.json
16:36 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1178.eqiad.wmnet with reason: Maintenance
16:36 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1178.eqiad.wmnet with reason: Maintenance
16:36 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T303603)', diff saved to https://phabricator.wikimedia.org/P28237 and previous config saved to /var/cache/conftool/dbconfig/20220521-163631-ladsgroup.json
16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1177 (T303603)', diff saved to https://phabricator.wikimedia.org/P28236 and previous config saved to /var/cache/conftool/dbconfig/20220521-160624-ladsgroup.json
16:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1177.eqiad.wmnet with reason: Maintenance
16:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1177.eqiad.wmnet with reason: Maintenance
16:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T303603)', diff saved to https://phabricator.wikimedia.org/P28235 and previous config saved to /var/cache/conftool/dbconfig/20220521-160616-ladsgroup.json
15:06 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1167 (T303603)', diff saved to https://phabricator.wikimedia.org/P28234 and previous config saved to /var/cache/conftool/dbconfig/20220521-150602-ladsgroup.json
15:06 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:06 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
15:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance
15:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1167.eqiad.wmnet with reason: Maintenance
15:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1126 (T303603)', diff saved to https://phabricator.wikimedia.org/P28233 and previous config saved to /var/cache/conftool/dbconfig/20220521-150549-ladsgroup.json
14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1126 (T303603)', diff saved to https://phabricator.wikimedia.org/P28232 and previous config saved to /var/cache/conftool/dbconfig/20220521-143507-ladsgroup.json
14:35 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance
14:35 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance
14:35 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1114 (T303603)', diff saved to https://phabricator.wikimedia.org/P28231 and previous config saved to /var/cache/conftool/dbconfig/20220521-143459-ladsgroup.json
14:28 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28230 and previous config saved to /var/cache/conftool/dbconfig/20220521-142836-ladsgroup.json
14:28 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db1105.eqiad.wmnet with reason: Maintenance
14:28 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 16:00:00 on db1105.eqiad.wmnet with reason: Maintenance
14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28229 and previous config saved to /var/cache/conftool/dbconfig/20220521-141926-ladsgroup.json
14:19 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1146.eqiad.wmnet with reason: Maintenance
14:19 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1146.eqiad.wmnet with reason: Maintenance
14:19 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298555)', diff saved to https://phabricator.wikimedia.org/P28228 and previous config saved to /var/cache/conftool/dbconfig/20220521-141918-ladsgroup.json
14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1114 (T303603)', diff saved to https://phabricator.wikimedia.org/P28227 and previous config saved to /var/cache/conftool/dbconfig/20220521-140520-ladsgroup.json
14:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1114.eqiad.wmnet with reason: Maintenance
14:05 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1114.eqiad.wmnet with reason: Maintenance
14:05 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1111 (T303603)', diff saved to https://phabricator.wikimedia.org/P28226 and previous config saved to /var/cache/conftool/dbconfig/20220521-140512-ladsgroup.json
13:34 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1111 (T303603)', diff saved to https://phabricator.wikimedia.org/P28225 and previous config saved to /var/cache/conftool/dbconfig/20220521-133431-ladsgroup.json
13:34 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1111.eqiad.wmnet with reason: Maintenance
13:34 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1111.eqiad.wmnet with reason: Maintenance
13:08 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1116.eqiad.wmnet with reason: Maintenance
13:08 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1116.eqiad.wmnet with reason: Maintenance
12:41 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
12:41 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance
12:41 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T303603)', diff saved to https://phabricator.wikimedia.org/P28224 and previous config saved to /var/cache/conftool/dbconfig/20220521-124124-ladsgroup.json
12:22 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T306560)', diff saved to https://phabricator.wikimedia.org/P28223 and previous config saved to /var/cache/conftool/dbconfig/20220521-122241-ladsgroup.json
12:20 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1164 (T306560)', diff saved to https://phabricator.wikimedia.org/P28222 and previous config saved to /var/cache/conftool/dbconfig/20220521-122023-ladsgroup.json
12:20 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
12:20 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
12:09 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3318 (T303603)', diff saved to https://phabricator.wikimedia.org/P28221 and previous config saved to /var/cache/conftool/dbconfig/20220521-120926-ladsgroup.json
12:09 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
12:09 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
11:59 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T298555)', diff saved to https://phabricator.wikimedia.org/P28220 and previous config saved to /var/cache/conftool/dbconfig/20220521-115919-ladsgroup.json
11:59 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1147.eqiad.wmnet with reason: Maintenance
11:59 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 10:00:00 on db1147.eqiad.wmnet with reason: Maintenance
11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance
11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance
11:43 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2079.codfw.wmnet with reason: Maintenance
11:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2079.codfw.wmnet with reason: Maintenance
11:43 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T303603)', diff saved to https://phabricator.wikimedia.org/P28219 and previous config saved to /var/cache/conftool/dbconfig/20220521-114318-ladsgroup.json
11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3318 (T303603)', diff saved to https://phabricator.wikimedia.org/P28218 and previous config saved to /var/cache/conftool/dbconfig/20220521-111146-ladsgroup.json
11:11 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
11:11 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
11:11 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1109 (T303603)', diff saved to https://phabricator.wikimedia.org/P28217 and previous config saved to /var/cache/conftool/dbconfig/20220521-111138-ladsgroup.json
10:42 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1109 (T303603)', diff saved to https://phabricator.wikimedia.org/P28216 and previous config saved to /var/cache/conftool/dbconfig/20220521-104247-ladsgroup.json
10:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1109.eqiad.wmnet with reason: Maintenance
10:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1109.eqiad.wmnet with reason: Maintenance
10:18 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance
10:18 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance
09:52 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
09:52 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
09:50 ladsgroup@cumin1001: END (PASS