You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(mutante: wb2-grrrri was not running and wikibugs had no more Gerrit updates since a while)
imported>Stashbot
(cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0))
 
(793 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== 2020-09-01 ==
== 2023-02-03 ==
* 00:02 mutante: wb2-grrrri was not running and wikibugs had no more Gerrit updates since a while
* 21:05 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 00:01 mutante: restarting wikibugs
* 21:04 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 21:02 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 21:00 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 20:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 20:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 19:44 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet
* 19:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1090.eqiad.wmnet with OS bullseye
* 19:00 dzahn@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "test what is not synced - dzahn@cumin2002"
* 18:59 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test what is not synced - dzahn@cumin2002"
* 18:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage
* 18:49 topranks: Enabling 4x10G channelization for pic 0 QSFP 4 on cr1-codfw
* 18:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage
* 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1090.eqiad.wmnet with OS bullseye
* 18:23 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet
* 18:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS bullseye
* 17:57 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage
* 17:57 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage
* 17:39 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet
* 17:36 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1089.eqiad.wmnet with OS bullseye
* 17:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1088.eqiad.wmnet with OS bullseye
* 17:34 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet
* 17:34 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1086.eqiad.wmnet with OS bullseye
* 17:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage
* 17:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1086.eqiad.wmnet with OS bullseye
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1089.eqiad.wmnet with OS bullseye
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 16:44 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
* 16:41 cmooney@cumin1001: START - Cookbook sre.dns.netbox
* 16:32 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2012.codfw.wmnet
* 16:25 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2012.codfw.wmnet
* 15:51 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): test (duration: 00m 26s)
* 15:51 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): test
* 15:23 milimetric@deploy1002: Finished deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection (duration: 00m 15s)
* 15:22 milimetric@deploy1002: Started deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection
* 14:31 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 09s)
* 14:31 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
* 14:20 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2011.codfw.wmnet
* 14:19 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 23s)
* 14:18 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
* 14:13 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2011.codfw.wmnet
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=ats-be
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=cdn
* 13:51 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1087.eqiad.wmnet with OS bullseye
* 13:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage
* 13:25 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage
* 13:05 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1087.eqiad.wmnet with OS bullseye
* 12:09 moritzm: installing node-moment security updates
* 12:01 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 13s)
* 12:00 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
* 11:58 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2010.codfw.wmnet
* 11:58 moritzm: installing node-qs security updates
* 11:50 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2010.codfw.wmnet
* 11:35 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2009.codfw.wmnet
* 11:28 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2009.codfw.wmnet
* 10:44 moritzm: updating perf on buster hosts
* 10:24 stevemunene@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:11 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2008.codfw.wmnet
* 10:07 stevemunene@cumin1001: END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:06 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:03 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2008.codfw.wmnet
* 09:51 moritzm: installing ruby-rack security updates
* 09:31 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:31 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:24 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:23 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:23 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:19 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet
* 09:14 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet
* 09:13 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:13 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 09:07 moritzm: installing modsecurity-crs security updates
* 09:02 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 09:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet
* 05:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1084.eqiad.wmnet with OS bullseye
* 05:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1085.eqiad.wmnet with OS bullseye
* 04:50 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage
* 04:47 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1084.eqiad.wmnet with OS bullseye
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1085.eqiad.wmnet with OS bullseye
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1083.eqiad.wmnet with OS bullseye
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1082.eqiad.wmnet with OS bullseye
* 03:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage
* 03:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1082.eqiad.wmnet with OS bullseye
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1083.eqiad.wmnet with OS bullseye
* 03:20 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet
* 03:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1080.eqiad.wmnet with OS bullseye
* 02:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage
* 02:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=ats-be
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=cdn
* 02:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1081.eqiad.wmnet with OS bullseye
* 02:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye
* 02:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage
* 02:00 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage
* 01:38 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS bullseye
* 01:31 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye
* 00:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye


== 2020-08-31 ==
== 2023-02-02 ==
* 23:38 crusnov@deploy1001: Finished deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 (final) (duration: 00m 17s)
* 22:58 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye
* 23:38 crusnov@deploy1001: Started deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 (final)
* 22:15 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet
* 23:37 crusnov@deploy1001: Finished deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox2001 (duration: 01m 12s)
* 22:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS bullseye
* 23:36 crusnov@deploy1001: Started deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox2001
* 22:01 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye
* 23:36 crusnov@deploy1001: Finished deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox1001 (duration: 00m 58s)
* 22:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet
* 23:35 crusnov@deploy1001: Started deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox1001
* 21:58 zabe@deploy1002: Finished scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] (duration: 07m 58s)
* 23:31 crusnov@deploy1001: Finished deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next pt2 (duration: 00m 05s)
* 21:52 zabe@deploy1002: zabe: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 23:31 crusnov@deploy1001: Started deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next pt2
* 21:50 zabe@deploy1002: Started scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]]
* 23:31 crusnov@deploy1001: Finished deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next (duration: 00m 57s)
* 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1078.eqiad.wmnet with OS bullseye
* 23:30 crusnov@deploy1001: Started deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next
* 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage
* 23:09 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable (future) mw-reverted tag for all wikis except testwiki ([[phab:T254074|T254074]]) (duration: 00m 57s)
* 21:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage
* 21:06 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:30 brennen: end of utc late backport & config window
* 21:00 pt1979@cumin2001: START - Cookbook sre.dns.netbox
* 21:30 brennen@deploy1002: Finished scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] (duration: 11m 14s)
* 20:20 ryankemper: `sudo systemctl restart elasticsearch_6@production-search-psi-eqiad.service` on `elastic1052.eqiad.wmnet`
* 21:23 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage
* 18:38 Urbanecm: Morning B&C done
* 21:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS bullseye
* 18:32 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|16197aabc88f098568a04984a20149de3b7fdeaf}}: Add two domains to wgCopyUploadsDomains for commonswiki ([[phab:T261562|T261562]]; [[phab:T261575|T261575]]) (duration: 00m 54s)
* 21:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
* 18:27 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|bb28e9da8057a4c92cd4d564ffd000f320338cda}}: itwiki: Assign patrol right to autopatrolled instead of autoconfirmed ([[phab:T261587|T261587]]) (duration: 00m 53s)
* 21:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1077.eqiad.wmnet with OS bullseye
* 18:23 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|a1b0d6e4e7da9bf45ae7381d2c1d9814e6b36498}}: {{Gerrit|b609cd53273e922cd8af5507660b9d10c6da09b3}}: CommonSettings.php: limit new Echos `push-subscription-manager` group to Meta-Wiki ([[phab:T261625|T261625]]) (duration: 00m 54s)
* 21:21 brennen@deploy1002: brennen and nray: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 18:11 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|846c5448f950b4d0d7eedce570e46d74ca62ca38}}: wgEventStreams: Stream for MEP-iOS pilot ([[phab:T260382|T260382]]) (duration: 00m 55s)
* 21:20 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage
* 17:21 volans: uploaded spicerack_0.0.42 to apt.wikimedia.org buster-wikimedia
* 21:19 brennen@deploy1002: Started scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]]
* 15:50 rzl@cumin1001: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
* 21:18 brennen@deploy1002: Finished scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] (duration: 12m 02s)
* 15:49 ejegg: updated payments-wiki from {{Gerrit|ef7ebd08cb}} to {{Gerrit|be81063168}}
* 21:07 brennen@deploy1002: brennen and dreamyjazz: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 15:34 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0)
* 21:06 brennen@deploy1002: Started scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]]
* 15:33 rzl@cumin1001: START - Cookbook sre.switchdc.services.02-restore-ttl
* 20:59 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye
* 15:33 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=99)
* 20:59 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1078.eqiad.wmnet with OS bullseye
* 15:32 rzl@cumin1001: START - Cookbook sre.switchdc.services.02-restore-ttl
* 20:52 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage
* 14:58 ema: Traffic: depool eqiad from user traffic [[phab:T243316|T243316]]
* 20:49 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage
* 14:38 moritzm: installing rake security updates on stretch
* 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye
* 14:33 rzl@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
* 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1077.eqiad.wmnet with OS bullseye
* 14:21 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0)
* 20:23 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include bullseye-wikimedia /home/rzl/httpbb/bullseye/httpbb_0.0.3-1+deb11u1_amd64.changes  # [[phab:T328280|T328280]]
* 14:20 rzl@cumin1001: Switching services apertium, termbox, search, api-gateway, ores, sessionstore, eventgate-main, graphoid, eventstreams, wikifeeds, wdqs, parsoid, eventgate-logging-external, wdqs-internal, echostore, mathoid, mobileapps, proton, restbase, kartotherian, recommendation-api, eventgate-analytics-external, restbase-async, citoid, schema, cxserver, eventgate-analytics, zotero: eqiad => codfw
* 20:21 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include buster-wikimedia /home/rzl/httpbb/buster/httpbb_0.0.3-1_amd64.changes  # [[phab:T328280|T328280]]
* 14:20 rzl@cumin1001: START - Cookbook sre.switchdc.services.01-switch-dc
* 20:11 zabe@deploy1002: Finished scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] (duration: 09m 39s)
* 14:18 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0)
* 20:03 zabe@deploy1002: zabe: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 14:13 rzl@cumin1001: START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep
* 20:02 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host elastic2037.codfw.wmnet
* 14:12 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=99)
* 20:01 zabe@deploy1002: Started scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]]
* 14:11 rzl@cumin1001: START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep
* 19:55 bking@cumin1001: START - Cookbook sre.hosts.reboot-single for host elastic2037.codfw.wmnet
* 13:41 andrewbogott: dropping many databases from m5, as per [[phab:T261152|T261152]]
* 19:54 ryankemper: [[phab:T328674|T328674]] [Elastic] With puppet disabled on elastic* fleet, `ryankemper@elastic2037:~$ sudo run-puppet-agent --force` to verify changes in https://gerrit.wikimedia.org/r/886055
* 13:15 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 19:30 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]]
* 13:15 jmm@cumin2001: START - Cookbook sre.hosts.downtime
* 19:28 zabe@deploy1002: say aborted:  (duration: 00m 03s)
* 13:07 marostegui: Failover m3 (phabricator) proxy from dbproxy1016 to dbproxy1020 - [[phab:T261459|T261459]]
* 18:42 zabe@deploy1002: Finished scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] (duration: 08m 19s)
* 13:05 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 18:36 zabe@deploy1002: zabe: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 13:02 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 18:34 zabe@deploy1002: Started scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]]
* 12:54 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0)
* 18:08 aokoth@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0
* 12:54 oblivian@cumin2001: START - Cookbook sre.switchdc.services.02-restore-ttl
* 18:08 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 12:53 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0)
* 18:08 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 12:53 oblivian@cumin2001: Switching services parsoid: eqiad => codfw
* 18:08 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2043.codfw.wmnet with OS bullseye
* 12:53 oblivian@cumin2001: START - Cookbook sre.switchdc.services.01-switch-dc
* 18:07 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 12:53 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0)
* 18:06 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 12:48 oblivian@cumin2001: START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep
* 18:05 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 12:45 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0)
* 18:05 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 12:45 oblivian@cumin2001: START - Cookbook sre.switchdc.services.02-restore-ttl
* 18:03 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1037.eqiad.wmnet with OS bullseye
* 12:44 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0)
* 17:52 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage
* 12:44 oblivian@cumin2001: Switching services restbase-async: eqiad => codfw
* 17:49 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage
* 12:44 oblivian@cumin2001: START - Cookbook sre.switchdc.services.01-switch-dc
* 17:47 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage
* 12:43 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0)
* 17:45 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage
* 12:37 oblivian@cumin2001: START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep
* 17:33 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2043.codfw.wmnet with OS bullseye
* 12:14 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0)
* 17:32 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1037.eqiad.wmnet with OS bullseye
* 12:14 oblivian@cumin2001: START - Cookbook sre.switchdc.services.02-restore-ttl
* 17:29 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0
* 12:13 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0)
* 17:12 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync
* 12:13 oblivian@cumin2001: Switching services restbase-async: eqiad => codfw
* 17:12 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: sync
* 12:13 oblivian@cumin2001: START - Cookbook sre.switchdc.services.01-switch-dc
* 16:53 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 12:10 oblivian@cumin2001: END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0)
* 16:52 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 12:05 oblivian@cumin2001: START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep
* 16:51 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 11:58 elukey: reimage kafka-jumbo1001 to Buster
* 16:50 dancy@deploy1002: Installation of scap version "4.34.0" completed for 561 hosts
* 11:32 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: {{Gerrit|5d583d9550787a8e36c29ca841233615405fcb7e}}: Disable MediaSearch A/B test (duration: 00m 55s)
* 16:50 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
* 11:22 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|81f88fde2aad23a619047b1177a6188f51df11a9}}: Enable Signature button on Wikiproject for hywiki ([[phab:T261550|T261550]]) (duration: 00m 54s)
* 16:50 dancy@deploy1002: Installing scap version "4.34.0" for 561 hosts
* 11:22 jbond42: removing old hiera version 1 and 3 backends
* 16:50 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:17 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|b74893fecdaae599077daad5b1219ad3b9bc7fc9}}: Enable sitenotice on mobile for closed wikis ([[phab:T261357|T261357]]) (duration: 00m 56s)
* 16:49 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 11:02 volans: upgraded spicerack to 0.0.41 on cumin hosts
* 16:48 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 10:27 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:48 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 09:51 elukey: executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address)
* 16:47 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: sync
* 08:43 moritzm: installing bind9 security updates on stretch/buster (client-side tools/libs only)
* 16:46 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: sync
* 07:53 volans: uploaded spicerack_0.0.41 to apt.wikimedia.org buster-wikimedia
* 16:25 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2007.codfw.wmnet
* 07:30 moritzm: installing squid security updates
* 16:18 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 07:24 moritzm: installing openexr security updates on buster
* 16:17 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 07:12 marostegui: Sanitize jawikivoyage  on db2094:3325 and db1124:3325 [[phab:T260482|T260482]]
* 16:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2007.codfw.wmnet
* 06:30 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 16:17 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 06:27 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 16:16 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
* 06:06 elukey: reimage kafka-jumbo1005 to Debian Buster
* 16:16 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 05:21 marostegui: Reload haproxy on dbproxy1017 and dbproxy1021 to test db1128
* 16:15 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 16:10 volans: uploaded python3-wmflib_1.2.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia
* 16:10 dzahn@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0
* 15:40 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided) (duration: 07m 01s)
* 15:38 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:37 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:35 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:35 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 15:34 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0
* 15:33 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided)
* 15:24 jmm@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti3004
* 15:17 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti3004
* 15:06 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2006.codfw.wmnet
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002"
* 15:02 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002"
* 15:00 vgutierrez: rolling restart of varnish in cache::text - [[phab:T315676|T315676]]
* 14:59 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 14:59 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2006.codfw.wmnet
* 14:55 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 14:45 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 14:39 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 14:31 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2005.codfw.wmnet
* 14:29 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 14:25 moritzm: installing containerd security updates on codfw k8s nodes
* 14:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2005.codfw.wmnet
* 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=ats-be
* 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=cdn
* 13:10 kharlan:: Deployed security patch for [[phab:T328643|T328643]]
* 13:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1076.eqiad.wmnet with OS bullseye
* 13:04 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 13:03 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 13:03 kharlan:: Deployed security patch for [[phab:T328643|T328643]]
* 13:02 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 13:01 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2004.codfw.wmnet
* 13:00 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 12:55 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2004.codfw.wmnet
* 12:47 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage
* 12:47 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 12:46 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 12:44 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage
* 12:42 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 12:42 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 12:39 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 12:39 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 12:29 btullis@deploy1002: Finished deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade (duration: 00m 42s)
* 12:29 claime: Work ongoing on m2 and m3
* 12:29 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2003.codfw.wmnet
* 12:29 btullis@deploy1002: Started deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade
* 12:23 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1076.eqiad.wmnet with OS bullseye
* 12:22 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2003.codfw.wmnet
* 12:08 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 12:08 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 11:46 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 11:42 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:42 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:41 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:41 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:40 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:39 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:38 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:37 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:37 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-4.out # [[phab:T328634|T328634]] – made some progress then errored out again
* 11:32 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-3.out # [[phab:T328634|T328634]] – seemed to finish the first 20 pages and then go into an infinite loop, I Ctrl+Ced it
* 11:28 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-2.out # [[phab:T328634|T328634]] – another error but made more progress
* 11:23 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes.out # [[phab:T328634|T328634]] – failed quickly, details in task
* 11:22 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: sync
* 11:22 elukey@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: sync
* 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:02 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
* 10:27 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2002.codfw.wmnet
* 10:19 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2002.codfw.wmnet
* 10:17 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 10:11 moritzm: restarting FPM on mw canaries to pick up tiff security updates
* 10:04 moritzm: installing tiff security updates
* 09:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2001.codfw.wmnet
* 09:55 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync
* 09:54 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync
* 09:51 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2001.codfw.wmnet
* 09:40 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync
* 09:40 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: sync
* 09:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 398143
* 09:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 398143
* 09:16 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6
* 09:13 apergos: UTC morning backport and config training window done
* 09:13 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync
* 09:12 elukey@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-main: sync
* 09:11 elukey: roll restart of eventgate-main pods in wikikube eqiad/codfw to pick up new stream configs - [[phab:T328576|T328576]]
* 08:57 ariel@deploy1002: Finished scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] (duration: 10m 56s)
* 08:48 ariel@deploy1002: ariel and aishik: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 08:46 ariel@deploy1002: Started scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]]
* 08:39 jelto@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6
* 08:37 tgr@deploy1002: Finished scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] (duration: 14m 26s)
* 08:27 tgr@deploy1002: tgr: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 08:23 tgr@deploy1002: Started scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]]
* 06:17 kart_: Updated cxserver to 2023-02-02-004918-production ([[phab:T129470|T129470]], [[phab:T172035|T172035]], [[phab:T327842|T327842]])
* 06:16 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 06:15 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 06:13 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 06:12 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 06:09 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 06:09 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 04:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5024.eqsin.wmnet
* 03:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS bullseye
* 03:21 ejegg: payments-wiki upgraded from {{Gerrit|f20a2208}} to {{Gerrit|53d1a58d}}
* 02:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage
* 02:46 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage
* 02:14 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye
* 02:14 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5024.eqsin.wmnet with OS bullseye
* 01:56 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye
* 01:55 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet
* 01:55 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5023.eqsin.wmnet with OS bullseye
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=cdn
* 01:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1075.eqiad.wmnet with OS bullseye
* 01:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage
* 01:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage
* 01:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage
* 01:18 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage
* 01:07 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1075.eqiad.wmnet with OS bullseye
* 00:44 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5023.eqsin.wmnet with OS bullseye
* 00:06 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5022.eqsin.wmnet
* 00:04 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS bullseye


== 2020-08-30 ==
== 2023-02-01 ==
* 16:13 herron: restarted eqiad v5 logstashes
* 23:45 zabe@deploy1002: Finished scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] (duration: 08m 07s)
* 23:39 zabe@deploy1002: zabe: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
* 23:37 zabe@deploy1002: Started scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]]
* 23:31 rzl@cumin2002: dbctl commit (dc=all): 'Depool db2181', diff saved to https://phabricator.wikimedia.org/P43574 and previous config saved to /var/cache/conftool/dbconfig/20230201-233140-rzl.json
* 23:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage
* 23:27 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage
* 23:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: security release
* 23:17 dancy@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]] (duration: 06m 57s)
* 23:10 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]]
* 23:01 zabe@deploy1002: Finished scap: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] (duration: 07m 45s)
* 22:55 zabe@deploy1002: zabe: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
* 22:54 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS bullseye
* 22:53 zabe@deploy1002: Started scap: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]]
* 22:49 zabe@deploy1002: Finished scap: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] (duration: 13m 03s)
* 22:47 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release
* 22:40 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS bullseye
* 22:38 zabe@deploy1002: zabe: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 22:36 zabe@deploy1002: Started scap: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]]
* 22:32 kindrobot: close UTC late backport window
* 22:31 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] (duration: 10m 37s)
* 22:22 kindrobot@deploy1002: nray and kindrobot: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 22:21 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]]
* 22:14 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] (duration: 18m 14s)
* 21:57 kindrobot@deploy1002: kindrobot and sbailey: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 21:57 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore100*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 21:56 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]]
* 21:53 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 21:52 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] (duration: 14m 53s)
* 21:43 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS bullseye
* 21:39 eevans@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore100*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 21:39 kindrobot@deploy1002: dreamyjazz and kindrobot: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 21:37 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]]
* 21:32 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] (duration: 13m 56s)
* 21:26 eevans@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=sessionstore,name=codfw
* 21:26 eevans@puppetmaster1001: conftool action : get/pooled=true; selector: dnsdisc=sessionstore,name=codfw
* 21:26 eevans@puppetmaster1001: conftool action : get/pooled=true; selector: dnsdisc=sessionstore,name=codfw
* 21:24 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release
* 21:20 kindrobot@deploy1002: arlolra and kindrobot: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
* 21:19 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 21:18 kindrobot@deploy1002: Started scap: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]]
* 21:14 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3065.esams.wmnet
* 21:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3065.esams.wmnet with OS bullseye
* 21:03 kindrobot: start UTC late backport deployment window
* 21:02 eevans@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001
* 20:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3065.esams.wmnet with reason: host reimage
* 20:44 eevans@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=sessionstore,name=codfw
* 20:43 urandom: depooling sessionstore —codfw— in preparation for Cassandra restarts — [[phab:T327675|T327675]]
* 20:42 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3065.esams.wmnet with reason: host reimage
* 20:40 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3064.esams.wmnet
* 20:38 eevans@puppetmaster1001: conftool action : get/pooled; selector: dnsdisc=$SERVICE,name=$DC
* 20:33 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3064.esams.wmnet with OS bullseye
* 20:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3065.esams.wmnet with OS bullseye
* 20:21 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3063.esams.wmnet
* 20:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3064.esams.wmnet with reason: host reimage
* 20:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3063.esams.wmnet with OS bullseye
* 20:08 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3064.esams.wmnet with reason: host reimage
* 20:03 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5031.eqsin.wmnet,service=ats-be
* 20:03 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5031.eqsin.wmnet,service=cdn
* 20:00 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5031.eqsin.wmnet with OS bullseye
* 19:53 dancy: The train is blocked on [[phab:T328601|T328601]]
* 19:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3064.esams.wmnet with OS bullseye
* 19:49 dancy@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.20  refs [[phab:T325584|T325584]] (duration: 06m 36s)
* 19:49 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3062.esams.wmnet
* 19:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3062.esams.wmnet with OS bullseye
* 19:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3063.esams.wmnet with reason: host reimage
* 19:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3063.esams.wmnet with reason: host reimage
* 19:42 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.20  refs [[phab:T325584|T325584]]
* 19:41 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5021.eqsin.wmnet,service=ats-be
* 19:41 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5021.eqsin.wmnet,service=cdn
* 19:37 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5021.eqsin.wmnet with OS bullseye
* 19:33 dancy@deploy1002: deploy-promote aborted:  (duration: 11m 58s)
* 19:33 dancy@deploy1002: sync-file aborted: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]] (duration: 03m 38s)
* 19:30 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5031.eqsin.wmnet with reason: host reimage
* 19:29 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.21  refs [[phab:T325584|T325584]]
* 19:27 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5031.eqsin.wmnet with reason: host reimage
* 19:26 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3062.esams.wmnet with reason: host reimage
* 19:24 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3063.esams.wmnet with OS bullseye
* 19:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3061.esams.wmnet
* 19:24 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3062.esams.wmnet with reason: host reimage
* 19:17 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3061.esams.wmnet with OS bullseye
* 19:04 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5021.eqsin.wmnet with reason: host reimage
* 19:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3062.esams.wmnet with OS bullseye
* 19:02 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3060.esams.wmnet
* 19:02 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3060.esams.wmnet with OS bullseye
* 19:01 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5021.eqsin.wmnet with reason: host reimage
* 18:56 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3061.esams.wmnet with reason: host reimage
* 18:55 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye
* 18:55 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5031.eqsin.wmnet with OS bullseye
* 18:52 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3061.esams.wmnet with reason: host reimage
* 18:47 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye
* 18:46 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5031.eqsin.wmnet with OS bullseye
* 18:39 jbond@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts puppetmaster2003.codfw.wmnet
* 18:38 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3060.esams.wmnet with reason: host reimage
* 18:37 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye
* 18:35 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3060.esams.wmnet with reason: host reimage
* 18:32 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3061.esams.wmnet with OS bullseye
* 18:31 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3059.esams.wmnet
* 18:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3059.esams.wmnet with OS bullseye
* 18:29 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye
* 18:29 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts puppetmaster2003.codfw.wmnet
* 18:29 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5021.eqsin.wmnet with OS bullseye
* 18:22 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye
* 18:21 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp1075.eqiad.wmnet with reason: downtimed for idrac firmware testing
* 18:20 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on cp1075.eqiad.wmnet with reason: downtimed for idrac firmware testing
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet,service=ats-be
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet,service=cdn
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet,service=ats-be
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet,service=cdn
* 18:13 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3060.esams.wmnet with OS bullseye
* 18:13 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3058.esams.wmnet
* 18:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3058.esams.wmnet with OS bullseye
* 18:10 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS bullseye
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43573 and previous config saved to /var/cache/conftool/dbconfig/20230201-181036-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43572 and previous config saved to /var/cache/conftool/dbconfig/20230201-181031-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43571 and previous config saved to /var/cache/conftool/dbconfig/20230201-181024-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43570 and previous config saved to /var/cache/conftool/dbconfig/20230201-181016-root.json
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43569 and previous config saved to /var/cache/conftool/dbconfig/20230201-181011-root.json
* 18:06 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3059.esams.wmnet with reason: host reimage
* 18:03 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3059.esams.wmnet with reason: host reimage
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43568 and previous config saved to /var/cache/conftool/dbconfig/20230201-175531-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43567 and previous config saved to /var/cache/conftool/dbconfig/20230201-175526-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43566 and previous config saved to /var/cache/conftool/dbconfig/20230201-175519-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43565 and previous config saved to /var/cache/conftool/dbconfig/20230201-175511-root.json
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43564 and previous config saved to /var/cache/conftool/dbconfig/20230201-175506-root.json
* 17:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43563 and previous config saved to /var/cache/conftool/dbconfig/20230201-175446-root.json
* 17:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3058.esams.wmnet with reason: host reimage
* 17:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3058.esams.wmnet with reason: host reimage
* 17:41 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3059.esams.wmnet with OS bullseye
* 17:40 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3057.esams.wmnet
* 17:40 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3057.esams.wmnet with OS bullseye
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43562 and previous config saved to /var/cache/conftool/dbconfig/20230201-174026-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43561 and previous config saved to /var/cache/conftool/dbconfig/20230201-174021-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43560 and previous config saved to /var/cache/conftool/dbconfig/20230201-174015-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43559 and previous config saved to /var/cache/conftool/dbconfig/20230201-174007-root.json
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43558 and previous config saved to /var/cache/conftool/dbconfig/20230201-174001-root.json
* 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43557 and previous config saved to /var/cache/conftool/dbconfig/20230201-173941-root.json
* 17:39 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage
* 17:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43555 and previous config saved to /var/cache/conftool/dbconfig/20230201-172521-root.json
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43554 and previous config saved to /var/cache/conftool/dbconfig/20230201-172516-root.json
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43553 and previous config saved to /var/cache/conftool/dbconfig/20230201-172510-root.json
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43552 and previous config saved to /var/cache/conftool/dbconfig/20230201-172502-root.json
* 17:24 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43551 and previous config saved to /var/cache/conftool/dbconfig/20230201-172456-root.json
* 17:24 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43550 and previous config saved to /var/cache/conftool/dbconfig/20230201-172436-root.json
* 17:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3058.esams.wmnet with OS bullseye
* 17:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3056.esams.wmnet
* 17:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3056.esams.wmnet with OS bullseye
* 17:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3057.esams.wmnet with reason: host reimage
* 17:17 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5019.eqsin.wmnet with OS bullseye
* 17:15 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3057.esams.wmnet with reason: host reimage
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43549 and previous config saved to /var/cache/conftool/dbconfig/20230201-171016-root.json
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43548 and previous config saved to /var/cache/conftool/dbconfig/20230201-171011-root.json
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43547 and previous config saved to /var/cache/conftool/dbconfig/20230201-171005-root.json
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43546 and previous config saved to /var/cache/conftool/dbconfig/20230201-170957-root.json
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43545 and previous config saved to /var/cache/conftool/dbconfig/20230201-170951-root.json
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43544 and previous config saved to /var/cache/conftool/dbconfig/20230201-170931-root.json
* 16:57 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:57 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye
* 16:57 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3056.esams.wmnet with reason: host reimage
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43543 and previous config saved to /var/cache/conftool/dbconfig/20230201-165512-root.json
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43542 and previous config saved to /var/cache/conftool/dbconfig/20230201-165506-root.json
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43541 and previous config saved to /var/cache/conftool/dbconfig/20230201-165500-root.json
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43540 and previous config saved to /var/cache/conftool/dbconfig/20230201-165452-root.json
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3056.esams.wmnet with reason: host reimage
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43539 and previous config saved to /var/cache/conftool/dbconfig/20230201-165446-root.json
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3057.esams.wmnet with OS bullseye
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43538 and previous config saved to /var/cache/conftool/dbconfig/20230201-165426-root.json
* 16:42 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:42 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye
* 16:40 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43536 and previous config saved to /var/cache/conftool/dbconfig/20230201-164007-root.json
* 16:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43535 and previous config saved to /var/cache/conftool/dbconfig/20230201-164002-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43534 and previous config saved to /var/cache/conftool/dbconfig/20230201-163955-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43533 and previous config saved to /var/cache/conftool/dbconfig/20230201-163947-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43532 and previous config saved to /var/cache/conftool/dbconfig/20230201-163941-root.json
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43531 and previous config saved to /var/cache/conftool/dbconfig/20230201-163921-root.json
* 16:33 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:33 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3056.esams.wmnet with OS bullseye
* 16:31 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye
* 16:29 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage
* 16:26 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage
* 16:25 jynus: reloaded apache on mailman
* 16:25 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye
* 16:23 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 16:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 16:15 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
* 16:14 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
* 16:14 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
* 16:13 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
* 15:53 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye
* 15:51 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5019.eqsin.wmnet with OS bullseye
* 15:31 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye
* 14:56 sukhe: cp1075.eqiad.wmnet for idrac firmware upgrade testing
* 14:55 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=ats-be
* 14:55 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=cdn
* 14:52 awight: EU deployment window complete
* 14:48 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:48 awight@deploy1002: Finished scap: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] (duration: 08m 25s)
* 14:47 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:41 awight@deploy1002: elukey and awight: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2136 db2158 db2157 es2026 db2106 db2146 [[phab:T327404|T327404]]', diff saved to https://phabricator.wikimedia.org/P43530 and previous config saved to /var/cache/conftool/dbconfig/20230201-144152-root.json
* 14:40 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:40 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:40 awight@deploy1002: Started scap: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]]
* 14:39 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:39 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:37 awight@deploy1002: Finished scap: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] (duration: 09m 22s)
* 14:29 awight@deploy1002: jdrewniak and awight: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 14:28 awight@deploy1002: Started scap: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]]
* 14:26 awight@deploy1002: Finished scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] (duration: 09m 07s)
* 14:19 awight@deploy1002: awight and mlitn: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
* 14:17 awight@deploy1002: Started scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]]
* 14:11 awight@deploy1002: backport aborted:  (duration: 06m 09s)
* 14:11 awight@deploy1002: sync-world aborted: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] (duration: 03m 36s)
* 14:09 awight@deploy1002: mlitn and awight: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 14:07 awight@deploy1002: Started scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]]
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast3005.wikimedia.org
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3005.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 14:06 moritzm: updating perf on Bullseye hosts
* 14:05 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3005.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 13:55 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:51 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast3005.wikimedia.org
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5002.wikimedia.org
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 13:47 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 13:43 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 13:36 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast5002.wikimedia.org
* 13:21 moritzm: installing curl security updates on bullseye
* 13:00 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
* 12:59 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2003.codfw.wmnet
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 12:40 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 12:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 12:27 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2003.codfw.wmnet
* 12:16 jmm@cumin2002: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for testvm2002.codfw.wmnet: Renew puppet certificate - jmm@cumin2002
* 12:15 jmm@cumin2002: START - Cookbook sre.puppet.renew-cert for testvm2002.codfw.wmnet: Renew puppet certificate - jmm@cumin2002
* 11:29 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part III ([[phab:T308932|T308932]]) (duration: 06m 43s)
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2001.codfw.wmnet
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 11:24 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
* 11:22 jgiannelos@deploy1002: Finished deploy [kartotherian/deploy@e1ca693] (codfw): Allow stylesheets through CSP (duration: 01m 45s)
* 11:21 ladsgroup@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part II ([[phab:T308932|T308932]]) (duration: 07m 04s)
* 11:21 jmm@cumin2002: START - Cookbook sre.dns.netbox
* 11:20 jgiannelos@deploy1002: Started deploy [kartotherian/deploy@e1ca693] (codfw): Allow stylesheets through CSP
* 11:17 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2001.codfw.wmnet
* 11:17 jgiannelos@deploy1002: Finished deploy [kartotherian/deploy@e1ca693] (eqiad): Allow stylesheets through CSP (duration: 00m 51s)
* 11:16 jgiannelos@deploy1002: Started deploy [kartotherian/deploy@e1ca693] (eqiad): Allow stylesheets through CSP
* 11:14 ladsgroup@deploy1002: Synchronized wmf-config/ext-CirrusSearch.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part I ([[phab:T308932|T308932]]) (duration: 07m 04s)
* 11:01 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a8840b0] (duration: 01m 18s)
* 11:00 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a8840b0]
* 10:59 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0] (thin): Regular analytics weekly train THIN [analytics/refinery@a8840b0] (duration: 00m 05s)
* 10:59 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0] (thin): Regular analytics weekly train THIN [analytics/refinery@a8840b0]
* 10:58 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0]: Regular analytics weekly train [analytics/refinery@a8840b0] (duration: 04m 29s)
* 10:54 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0]: Regular analytics weekly train [analytics/refinery@a8840b0]
* 10:52 steve_munene: Deploying refinery for ops week
* 10:42 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 10:42 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 10:42 zabe: start running migrateRevisionCommentTemp in remaining sections (for now except s3) in screens # [[phab:T275246|T275246]]
* 10:42 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 10:42 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 10:41 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 10:41 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 10:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb2002.codfw.wmnet with OS bullseye
* 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb2002.codfw.wmnet with reason: host reimage
* 10:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on krb2002.codfw.wmnet with reason: host reimage
* 10:01 godog: upgrade grafana to 8.5.20 on cloudmetrics* - [[phab:T328405|T328405]]
* 09:57 godog: upgrade grafana to 8.5.20 on grafana1002 - [[phab:T328405|T328405]]
* 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host krb2002.codfw.wmnet with OS bullseye
* 09:47 godog: upgrade grafana to 8.5.20 on grafana2001 - [[phab:T328405|T328405]]
* 09:15 urbanecm: Clean sign up throttle for IP 195.113.145.2 (via resetAuthenticationThrottle.php; [[phab:T328521|T328521]])
* 09:14 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:885734{{!}}Add new throttle rule (T328521)]] (duration: 07m 24s)
* 09:07 urbanecm@deploy1002: Started scap: Backport for [[gerrit:885734{{!}}Add new throttle rule (T328521)]]
* 09:06 urbanecm@deploy1002: backport aborted:  (duration: 00m 01s)
* 09:05 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] (duration: 15m 06s)
* 08:54 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: apply on main
* 08:54 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
* 08:52 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet
* 08:50 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]]
* 08:49 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] (duration: 08m 05s)
* 08:45 jayme@cumin1001: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=k8s-ingress-staging
* 08:45 jayme@cumin1001: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=k8s-ingress-staging
* 08:42 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
* 08:41 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]]
* 08:40 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] (duration: 12m 26s)
* 08:29 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
* 08:27 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]]
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
* 08:27 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] (duration: 09m 42s)
* 08:19 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 6 hosts
* 08:19 jayme@cumin1001: START - Cookbook sre.hosts.remove-downtime for 6 hosts
* 08:19 ladsgroup@deploy1002: ladsgroup and krinkle: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
* 08:17 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]]
* 08:14 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] (duration: 10m 15s)
* 08:06 ladsgroup@deploy1002: phedenskog and ladsgroup: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet
* 08:05 moritzm: installing libarchive security updates
* 08:04 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]]
* 08:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 55821
* 07:57 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 55821
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43524 and previous config saved to /var/cache/conftool/dbconfig/20230201-073348-ladsgroup.json
* 07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P43523 and previous config saved to /var/cache/conftool/dbconfig/20230201-071841-ladsgroup.json
* 07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P43522 and previous config saved to /var/cache/conftool/dbconfig/20230201-070335-ladsgroup.json
* 06:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43521 and previous config saved to /var/cache/conftool/dbconfig/20230201-064828-ladsgroup.json
* 06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43520 and previous config saved to /var/cache/conftool/dbconfig/20230201-064311-ladsgroup.json
* 06:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 06:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 06:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 06:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 00:38 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3055.esams.wmnet
* 00:37 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3055.esams.wmnet with OS bullseye
* 00:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3055.esams.wmnet with reason: host reimage
* 00:12 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3055.esams.wmnet with reason: host reimage
* 00:02 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3054.esams.wmnet
* 00:01 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3054.esams.wmnet with OS bullseye


== 2020-08-29 ==
==Archives ==
* 18:05 Amir1: end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https ([[phab:T261451|T261451]])
* 17:45 Amir1: start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https ([[phab:T261451|T261451]])
 
== 2020-08-28 ==
* 21:53 ryankemper: `sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues
* 19:11 andrewbogott: rebooting cloudvirt1006.  It's a spare, unused system but showing a bus error and icinga alerts; not worth saving if it needs saving
* 17:39 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 17:39 mutante: shutting down mw2196
* 17:37 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 16:40 rzl: switchdc live test complete
* 16:36 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0)
* 16:35 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-update-tendril
* 16:35 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0)
* 16:34 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance
* 16:33 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0)
* 16:33 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 16:33 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99)
* 16:33 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 16:29 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0)
* 16:29 rzl@cumin1001: [DRY-RUN] MediaWiki read-only period ends at: 2020-08-28 16:29:24.432463
* 16:29 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.07-set-readwrite
* 16:29 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)
* 16:29 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
* 16:29 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0)
* 16:29 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions
* 16:29 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0)
* 16:29 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki
* 16:28 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0)
* 16:28 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
* 16:28 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0)
* 16:28 rzl@cumin1001: [DRY-RUN] MediaWiki read-only period starts at: 2020-08-28 16:28:07.882663
* 16:28 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.02-set-readonly
* 16:19 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0)
* 16:19 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance
* 16:13 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0)
* 16:12 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-warmup-caches
* 16:09 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0)
* 16:09 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 16:08 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99)
* 16:08 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 16:07 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0)
* 16:07 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-disable-puppet
* 16:06 rzl: starting one more live test of the data center switchover automation, no production impact is expected but there will be some SAL noise
* 14:22 moritzm: installing Java security updates on kafka/main and Logstash(5) clusters
* 13:35 hashar@deploy1001: Finished deploy [integration/docroot@65ec92c]: noop, sync up for README.md (duration: 00m 07s)
* 13:35 hashar@deploy1001: Started deploy [integration/docroot@65ec92c]: noop, sync up for README.md
* 13:25 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 13:23 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 13:07 elukey: stop kafka on kafka-jumbo1006 and reimage to buster
* 12:56 moritzm: installing debmonitor1002 [[phab:T261492|T261492]]
* 12:46 moritzm: installing debmonitor2002 [[phab:T261492|T261492]]
* 11:50 jmm@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 11:40 jmm@cumin2001: START - Cookbook sre.ganeti.makevm
* 11:37 jmm@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 11:27 jmm@cumin2001: START - Cookbook sre.ganeti.makevm
* 11:27 jmm@cumin2001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
* 11:27 jmm@cumin2001: START - Cookbook sre.ganeti.makevm
* 09:48 jayme: updated helm to 2.16.9-3 on chartmuseum*, contint*, deploy*
* 09:19 jayme: imported helm_2.16.9-3 to buster-wikimedia, stretch-wikimedia, jessie-wikimedia
* 08:22 kormat: enabling replication from db2112 to db1083 (s1) [[phab:T243373|T243373]]
* 07:41 jynus: restart backup2001,backup1002
* 07:10 jynus: restart db2139
* 07:07 marostegui: Warm up parsercache in codfw - [[phab:T260042|T260042]]
* 06:47 jynus: restart db2102
* 06:28 jynus: restart db2100
* 06:07 jynus: restart db2099
* 05:50 jynus: restart db2098
* 00:06 eileen: process-control config revision is {{Gerrit|dd541a25dc}}
 
== 2020-08-27 ==
* 23:53 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 23:51 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 23:48 eileen: civicrm revision changed from {{Gerrit|a942537984}} to {{Gerrit|3d501e71d9}}, config revision is {{Gerrit|dd541a25dc}}
* 22:54 eileen: civicrm revision changed from {{Gerrit|481ab742db}} to {{Gerrit|a942537984}}, config revision is {{Gerrit|e2ab4d7c1f}}
* 22:28 tzatziki: removing one file for legal compliance
* 22:18 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 22:18 volans: uploaded spicerack_0.0.40-1_amd64.deb to apt.wikimedia.org buster-wikimedia
* 22:17 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:59 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:57 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:46 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:43 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:43 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:41 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:34 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:33 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:32 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:29 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:28 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:25 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:25 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:23 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:23 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:22 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:22 dzahn@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97)
* 21:20 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:20 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:17 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:17 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
* 21:17 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:17 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:16 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:15 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99)
* 21:14 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:13 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 21:13 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:10 dzahn@cumin1001: END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97)
* 21:07 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 20:50 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 20:50 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 20:50 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=api_appserver,name=mw221[0-4].codfw.wmnet
* 20:49 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 20:49 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 20:49 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=api_appserver,name=mw220[0-9].codfw.wmnet
* 20:48 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 20:48 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 20:48 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=api_appserver,name=mw214[0-7].codfw.wmnet
* 20:48 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 20:48 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 20:47 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=api_appserver,name=mw213[0-9].codfw.wmnet
* 20:43 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Streams for testing MEP-based analytics instruments - [[phab:T259714|T259714]] (duration: 00m 55s)
* 19:58 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 19:58 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 19:58 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 19:58 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 19:57 marxarelli: 1.36.0-wmf.6 promoted to all wikis ([[phab:T257974|T257974]]). new errors appear to be related to [[phab:T261345|T261345]] but are known since 1.36.0-wmf.5
* 19:57 dzahn@cumin1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=appserver,name=mw21[8-9][0-9]*.codfw.wmnet
* 19:41 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.6
* 19:22 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 11s)
* 19:20 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating apiportalwiki ([[phab:T246945|T246945]]) (duration: 01m 03s)
* 19:19 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating apiportalwiki ([[phab:T246945|T246945]]) (duration: 01m 03s)
* 19:16 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating apiportalwiki ([[phab:T246945|T246945]])
* 19:15 urbanecm@deploy1001: Synchronized dblists: Creating apiportalwiki ([[phab:T246945|T246945]]) (duration: 01m 03s)
* 19:14 urbanecm@deploy1001: Synchronized multiversion/MWMultiVersion.php: Creating apiportalwiki ([[phab:T246945|T246945]]) (duration: 01m 03s)
* 19:13 urbanecm@deploy1001: Synchronized wmf-config/db-codfw.php: Creating apiportalwiki ([[phab:T246945|T246945]]) (duration: 01m 03s)
* 19:11 urbanecm@deploy1001: Synchronized wmf-config/db-eqiad.php: Creating apiportalwiki ([[phab:T246945|T246945]]) (duration: 01m 03s)
* 18:54 mforns@deploy1001: Finished deploy [analytics/refinery@e85191b] (thin): Regular analytics weekly train THIN [analytics/refinery@e85191bb80c13781b41031340ea318b5f854b6a9] (duration: 00m 08s)
* 18:54 mforns@deploy1001: Started deploy [analytics/refinery@e85191b] (thin): Regular analytics weekly train THIN [analytics/refinery@e85191bb80c13781b41031340ea318b5f854b6a9]
* 18:53 mforns@deploy1001: Finished deploy [analytics/refinery@e85191b]: Regular analytics weekly train [analytics/refinery@e85191bb80c13781b41031340ea318b5f854b6a9] (duration: 10m 01s)
* 18:43 mforns@deploy1001: Started deploy [analytics/refinery@e85191b]: Regular analytics weekly train [analytics/refinery@e85191bb80c13781b41031340ea318b5f854b6a9]
* 18:43 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Assign all homepage users to variant A (duration: 01m 03s)
* 18:31 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable GrowthExperiments on ruwiki ([[phab:T257490|T257490]]) (duration: 01m 03s)
* 18:17 dzahn@cumin1001: conftool action : set/weight=1; selector: dc=codfw,cluster=jobrunner,name=mw2250.codfw.wmnet,service=canary
* 18:17 dzahn@cumin1001: conftool action : set/weight=1; selector: dc=codfw,cluster=jobrunner,name=mw2249.codfw.wmnet,service=canary
* 18:16 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 18:16 cdanis@cumin1001: START - Cookbook sre.network.cf
* 18:14 dzahn@cumin1001: conftool action : set/weight=10; selector: dc=eqiad,cluster=jobrunner,name=mw1318.eqiad.wmnet
* 18:07 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw229[1-9].codfw.wmnet,cluster=api_appserver
* 18:06 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw2290.codfw.wmnet,cluster=api_appserver
* 18:05 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw22[6-8][0-9].codfw.wmnet,cluster=api_appserver
* 18:03 Urbanecm: Creating jawikivoyage is done ([[phab:T260320|T260320]])
* 18:02 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 01m 59s)
* 18:02 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw225[0-9].codfw.wmnet,cluster=api_appserver
* 18:00 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating jawikivoyage ([[phab:T260320|T260320]]) (duration: 01m 02s)
* 17:59 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw224[4-5].codfw.wmnet,service=canary
* 17:59 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw224[4-5].codfw.wmnet
* 17:59 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating jawikivoyage ([[phab:T260320|T260320]]) (duration: 01m 03s)
* 17:58 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating jawikivoyage ([[phab:T260320|T260320]])
* 17:57 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw222[0-3].codfw.wmnet
* 17:56 urbanecm@deploy1001: Synchronized dblists: Creating jawikivoyage ([[phab:T260320|T260320]]) (duration: 00m 58s)
* 17:56 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw221[5-9].codfw.wmnet,service=canary
* 17:55 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw221[5-9].codfw.wmnet
* 17:55 urbanecm@deploy1001: Synchronized wmf-config/db-codfw.php: Creating jawikivoyage ([[phab:T260320|T260320]]) (duration: 01m 03s)
* 17:54 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw221[0-4].codfw.wmnet
* 17:54 dzahn@cumin1001: conftool action : set/weight=10; selector: name=mw221[0-4].codfw.wmnet
* 17:54 urbanecm@deploy1001: Synchronized wmf-config/db-eqiad.php: Creating jawikivoyage ([[phab:T260320|T260320]]) (duration: 01m 07s)
* 17:52 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw220[1-9].codfw.wmnet
* 17:52 dzahn@cumin1001: conftool action : set/weight=10; selector: name=mw220[1-9].codfw.wmnet
* 17:51 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2200.codfw.wmnet
* 17:50 dzahn@cumin1001: conftool action : set/weight=10; selector: name=mw2200.codfw.wmnet
* 17:48 dzahn@cumin1001: conftool action : set/weight=10; selector: name=mw214[0-7].codfw.wmnet
* 17:47 dzahn@cumin1001: conftool action : set/weight=10; selector: name=mw213[5-9].codfw.wmnet
* 17:46 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw214[0-7].codfw.wmnet
* 17:45 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw213[5-9].codfw.wmnet
* 17:39 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw23[0-7][0-9].codfw.wmnet
* 17:31 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw227[0-7].codfw.wmnet,service=canary
* 17:30 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw227[0-7].codfw.wmnet
* 17:29 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 17:29 cdanis@cumin1001: START - Cookbook sre.network.cf
* 17:18 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:17 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw226[8-9].codfw.wmnet
* 17:13 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw225[4-8].codfw.wmnet
* 17:12 pt1979@cumin2001: START - Cookbook sre.dns.netbox
* 17:11 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw224[0-2].codfw.wmnet
* 17:04 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw223[2-9].codfw.wmnet
* 17:01 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw2231.codfw.wmnet
* 16:59 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw2230.codfw.wmnet
* 16:54 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw222[4-9].codfw.wmnet
* 16:49 mutante: re-weighted appservers and api appservers in eqiad - hardware type G = weight 25, all other types = weight 30 ([[phab:T261159|T261159]])
* 16:48 mutante: depooling mw2187 - mw2199 - old codfw appservers of type A to be decom'ed, previously weight 10 ([[phab:T260654|T260654]])
* 16:47 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw219[0-9].codfw.wmnet
* 16:41 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw218[7-9].codfw.wmnet
* 16:35 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw1297.eqiad.wmnet
* 16:23 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:21 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw127[0-5].eqiad.wmnet
* 16:19 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw126[1-5].eqiad.wmnet,service=canary
* 16:14 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw126[1-9].eqiad.wmnet
* 16:12 elukey: remove some old/stale terms from analytics-in4 on cr1/cr2-eqiad (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/622746, https://gerrit.wikimedia.org/r/c/operations/homer/public/+/622744)
* 16:09 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw127[6-9].eqiad.wmnet,service=canary
* 16:08 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw127[6-9].eqiad.wmnet
* 16:06 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw1290.eqiad.wmnet
* 16:05 dzahn@cumin1001: conftool action : set/weight=25; selector: name=mw128[0-9].eqiad.wmnet
* 15:52 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw1290.eqiad.wmnet
* 15:51 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw128[0-9].eqiad.wmnet
* 15:43 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw127[7-9].eqiad.wmnet,service=canary
* 15:43 dzahn@cumin1001: conftool action : set/weight=1; selector: name=mw1276.eqiad.wmnet,service=canary
* 15:41 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw127[6-9].eqiad.wmnet
* 15:39 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw1297.eqiad.wmnet
* 15:38 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw1269.eqiad.wmnet
* 15:38 dzahn@cumin1001: conftool action : set/weight=30; selector: name=mw1267.eqiad.wmnet
* 14:48 moritzm: installing Java security updates on aqs, hadoop and kafka-jumbo
* 14:44 moritzm: restarting tomcat on idp-test* hosts to pick up Java update
* 14:42 elukey: add eventgate-related terms to analytics-in4 filter on cr1/cr2-eqiad (ref https://gerrit.wikimedia.org/r/c/operations/homer/public/+/622705)
* 14:37 moritzm: imported openjdk 8u265-b01-1~deb10u1 to buster-wikimedia (forward port of latest Java 8 security update)
* 14:31 papaul: replacing msw-c5,c6,c7 and fmsw-c8
* 13:58 kormat: disabling GTID on pc2007 (pc1), pc2008 (pc2), pc2009 (pc3) [[phab:T243373|T243373]]
* 13:56 kormat: disabling GTID on db2096 (x1), es2021 (es4), es2023 (es5) [[phab:T243373|T243373]]
* 13:54 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 13:53 kormat: disabling GTID on db2129 (s6), db2118 (s7), db2079 (s8) [[phab:T243373|T243373]]
* 13:52 kormat: disabling GTID on db2123 (s5) [[phab:T243373|T243373]]
* 13:52 kormat: disabling GTID on db2090 (s4) [[phab:T243373|T243373]]
* 13:51 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 13:51 kormat: disabling GTID on db2105 (s3) [[phab:T243373|T243373]]
* 13:50 kormat: disabling GTID on db2107 (s2) [[phab:T243373|T243373]]
* 13:50 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 13:29 elukey: restart jvm daemons on analytics1042, aqs1004, kafka-jumbo1001 to pick up new openjdk upgrades (canaries)
* 13:18 kormat: enabling replication from db2107 to db1122 (s2) [[phab:T243373|T243373]]
* 13:14 kormat: enabling replication from db2096 to db1103 (x1) [[phab:T243373|T243373]]
* 13:10 jynus: restart db2097
* 13:07 jbond42: deploy python3.4 security update to kraz
* 13:03 jbond42: deploy python3.4 security update to canaries on jessie
* 13:01 kormat: enabling replication from db2118 to db1086 (s7) [[phab:T243373|T243373]]
* 12:52 jynus: restart db1140
* 12:43 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s8 weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12402 and previous config saved to /var/cache/conftool/dbconfig/20200827-124338-marostegui.json
* 12:35 jynus: restart db1139
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s7  weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12401 and previous config saved to /var/cache/conftool/dbconfig/20200827-123028-marostegui.json
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s7  weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12400 and previous config saved to /var/cache/conftool/dbconfig/20200827-123003-marostegui.json
* 12:24 marostegui: Fix password format for in db2129 (s6 codfw master) [[phab:T243373|T243373]]
* 12:14 kormat: enabling replication from db2129 to db1093 (s6) [[phab:T243373|T243373]]
* 12:13 jynus: restart db1095
* 12:08 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s6  weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12399 and previous config saved to /var/cache/conftool/dbconfig/20200827-120816-marostegui.json
* 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s5 codfw weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12398 and previous config saved to /var/cache/conftool/dbconfig/20200827-120211-marostegui.json
* 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s5 eqiad weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12397 and previous config saved to /var/cache/conftool/dbconfig/20200827-115934-marostegui.json
* 11:56 Urbanecm: Lift range blocks exceeding wgBlockCIDRLimit via custom script from F32197596 (ruwiki, ruwikiquote; [[phab:T243980|T243980]])
* 11:51 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s4 codfw weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12396 and previous config saved to /var/cache/conftool/dbconfig/20200827-115110-marostegui.json
* 11:49 moritzm: uploaded python3.4 3.4.2-1+deb8u7+wmf1 for jessie-wikimedia [[phab:T259102|T259102]]
* 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust s1 codfw weights [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12395 and previous config saved to /var/cache/conftool/dbconfig/20200827-114509-marostegui.json
* 11:22 marostegui@cumin1001: dbctl commit (dc=all): 'Adjust db2126 weight [[phab:T243373|T243373]]', diff saved to https://phabricator.wikimedia.org/P12394 and previous config saved to /var/cache/conftool/dbconfig/20200827-112213-marostegui.json
* 11:12 Urbanecm: EU B&C done
* 11:09 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|34994d39f92b23934929c66f3e15aa332683e746}}: Add $wgTranslateMessageNamespaces[] = NS_MEDIAWIKI; for commonswiki ([[phab:T131300|T131300]]) (duration: 01m 03s)
* 10:57 mvolz@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'zotero' for release 'production' .
* 10:56 godog: bounce grafana to apply new settings
* 10:51 kormat: enabling replication from db2123 to db1100 (s5) [[phab:T243373|T243373]]
* 10:48 mvolz@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'zotero' for release 'production' .
* 10:30 kormat: enabling replication from es2023 to es1024 (es5) [[phab:T243373|T243373]]
* 10:28 mvolz@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' .
* 10:23 kormat: enabling replication from es2021 to es1021 (es4) [[phab:T243373|T243373]]
* 10:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 10:19 ayounsi@cumin1001: START - Cookbook sre.network.cf
* 10:03 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 10:03 ayounsi@cumin1001: START - Cookbook sre.network.cf
* 09:54 moritzm: installing Java security updates on IDP* hosts
* 09:51 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 09:47 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 09:45 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 09:44 elukey@cumin1001: START - Cookbook sre.hosts.decommission
* 09:44 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 09:43 elukey@cumin1001: START - Cookbook sre.hosts.decommission
* 09:43 elukey: decommissioning vms schema[12]00[12] (replaced previously by schema[12]00[34] buster vms)
* 09:42 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 09:41 elukey@cumin1001: START - Cookbook sre.hosts.decommission
* 09:40 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 09:39 elukey@cumin1001: START - Cookbook sre.hosts.decommission
* 09:20 kormat: enabling replication from db2105 to db1123 (s3) [[phab:T243373|T243373]]
* 09:15 kormat: enabling replication from db2079 to db1109 (s8) [[phab:T243373|T243373]]
* 09:07 kormat: enabling replication from db2090 to db1081 (s4) [[phab:T243373|T243373]]
* 08:53 kormat: enabling replication from pc2009 to pc1009 (pc3) [[phab:T243373|T243373]]
* 08:44 kormat: enabling replication from pc2008 to pc1008 (pc2) [[phab:T243373|T243373]]
* 08:13 marostegui: Enable replication codfw -> eqiad on pc1 [[phab:T243373|T243373]]
* 08:01 gehel: manual cleanup of stale wdqs deploy crontab on wdqs1009
* 07:35 marostegui: Move pc2010 under pc2007 [[phab:T243373|T243373]]
* 07:16 moritzm: installing ghostscript security updates on stretch
* 06:50 elukey@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 06:49 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 06:46 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 06:45 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 06:34 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 06:31 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 06:06 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12392 and previous config saved to /var/cache/conftool/dbconfig/20200827-060652-marostegui.json
* 05:58 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12391 and previous config saved to /var/cache/conftool/dbconfig/20200827-055815-marostegui.json
* 05:55 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12390 and previous config saved to /var/cache/conftool/dbconfig/20200827-055522-marostegui.json
* 05:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079', diff saved to https://phabricator.wikimedia.org/P12389 and previous config saved to /var/cache/conftool/dbconfig/20200827-055126-marostegui.json
* 05:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12388 and previous config saved to /var/cache/conftool/dbconfig/20200827-055104-marostegui.json
* 05:43 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P12387 and previous config saved to /var/cache/conftool/dbconfig/20200827-054259-marostegui.json
* 05:41 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1074 db1085 db1078', diff saved to https://phabricator.wikimedia.org/P12386 and previous config saved to /var/cache/conftool/dbconfig/20200827-054114-marostegui.json
* 05:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P12385 and previous config saved to /var/cache/conftool/dbconfig/20200827-053814-marostegui.json
* 05:35 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12384 and previous config saved to /var/cache/conftool/dbconfig/20200827-053558-marostegui.json
* 05:35 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1074, db1085, db1079', diff saved to https://phabricator.wikimedia.org/P12383 and previous config saved to /var/cache/conftool/dbconfig/20200827-053509-marostegui.json
* 05:31 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1074, db1085, db1079', diff saved to https://phabricator.wikimedia.org/P12382 and previous config saved to /var/cache/conftool/dbconfig/20200827-053100-marostegui.json
* 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P12381 and previous config saved to /var/cache/conftool/dbconfig/20200827-052925-marostegui.json
* 05:28 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P12380 and previous config saved to /var/cache/conftool/dbconfig/20200827-052818-marostegui.json
* 05:24 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1074', diff saved to https://phabricator.wikimedia.org/P12379 and previous config saved to /var/cache/conftool/dbconfig/20200827-052413-marostegui.json
* 05:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P12378 and previous config saved to /var/cache/conftool/dbconfig/20200827-051609-marostegui.json
* 05:15 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12377 and previous config saved to /var/cache/conftool/dbconfig/20200827-051546-marostegui.json
* 05:07 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12376 and previous config saved to /var/cache/conftool/dbconfig/20200827-050754-marostegui.json
* 05:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1085', diff saved to https://phabricator.wikimedia.org/P12375 and previous config saved to /var/cache/conftool/dbconfig/20200827-050727-marostegui.json
* 04:53 marostegui: Stop db1074 and db2107 in sync to fix drifts on s2 change_tag - [[phab:T260042|T260042]]
* 04:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1074', diff saved to https://phabricator.wikimedia.org/P12374 and previous config saved to /var/cache/conftool/dbconfig/20200827-045329-marostegui.json
* 04:04 ryankemper@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1006.wikimedia.org
* 04:03 ryankemper@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1005.wikimedia.org
* 04:01 ryankemper@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cloudelastic1005.wikimedia.org
* 02:03 mutante: shutting down install3001,install4001,install5001 VMs (no OS yet, but please also don't delete, debugging in progress, shutting them down until I continue on [[phab:T254157|T254157]])
 
== 2020-08-26 ==
* 23:35 eileen: civicrm revision changed from {{Gerrit|d2e80f7522}} to {{Gerrit|481ab742db}}, config revision is {{Gerrit|e2ab4d7c1f}}
* 23:00 ryankemper@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
* 22:36 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 22:30 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 22:26 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
* 22:20 dzahn@cumin1001: START - Cookbook sre.ganeti.makevm
* 19:51 XioNoX: standardize pfw3-eqiad
* 19:33 marxarelli: 1.36.0-wmf.6 promoted to group1 ([[phab:T257974|T257974]]). logs show no new errors
* 19:24 dduvall@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.6 (duration: 01m 03s)
* 19:23 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.6
* 18:21 Urbanecm: Morning B&C done
* 18:20 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|945b97cff8b8a1e4bb43b613fc93b099f74945f7}}: Added import sources for mlwiktionary ([[phab:T260716|T260716]]) (duration: 01m 05s)
* 18:12 Urbanecm: Purge Thai and Greek taglines, URLs are at P12372  ([[phab:T258552|T258552]])
* 18:11 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|40092898d8c70191324e844d2c222469b954e9ef}}: Update Thai and Greek taglines ([[phab:T258552|T258552]]) (duration: 01m 03s)
* 18:09 urbanecm@deploy1001: Synchronized static/images/mobile/copyright/: {{Gerrit|40092898d8c70191324e844d2c222469b954e9ef}}: Update Thai and Greek taglines ([[phab:T258552|T258552]]) (duration: 01m 05s)
* 18:08 herron: upgraded eqiad elk v7 cluster from 7.8.0 to 7.9.0 [[phab:T234854|T234854]]
* 18:02 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 18:02 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
* 17:41 mholloway-shell@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable client side error logging on hewiki ([[phab:T255585|T255585]]) (duration: 01m 04s)
* 17:14 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Documentation-only change; sync for line sanity (duration: 01m 04s)
* 17:12 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: [[phab:T254349|T254349]] Set wgVisualEditorEnableBetaFeature true on wikis that need it (duration: 01m 03s)
* 15:59 oblivian@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' .
* 15:53 oblivian@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' .
* 15:41 oblivian@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' .
* 15:11 oblivian@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' .
* 14:56 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1134 for MCR change', diff saved to https://phabricator.wikimedia.org/P12371 and previous config saved to /var/cache/conftool/dbconfig/20200826-145612-marostegui.json
* 14:55 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1091 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12370 and previous config saved to /var/cache/conftool/dbconfig/20200826-145531-marostegui.json
* 14:47 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1091 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12369 and previous config saved to /var/cache/conftool/dbconfig/20200826-144750-marostegui.json
* 14:45 elukey@puppetmaster1001: conftool action : set/pooled=inactive:weight=0; selector: name=schema1002.eqiad.wmnet
* 14:45 elukey@puppetmaster1001: conftool action : set/pooled=inactive:weight=0; selector: name=schema1001.eqiad.wmnet
* 14:45 elukey@puppetmaster1001: conftool action : set/pooled=inactive:weight=0; selector: name=schema2001.codfw.wmnet
* 14:45 elukey@puppetmaster1001: conftool action : set/pooled=inactive:weight=0; selector: name=schema2002.codfw.wmnet
* 14:36 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1091 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12368 and previous config saved to /var/cache/conftool/dbconfig/20200826-143623-marostegui.json
* 14:34 elukey@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=schema2003.codfw.wmnet
* 14:34 elukey@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=schema2004.codfw.wmnet
* 14:33 elukey@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=schema1004.eqiad.wmnet
* 14:33 elukey@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: name=schema1003.eqiad.wmnet
* 14:27 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1091 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12367 and previous config saved to /var/cache/conftool/dbconfig/20200826-142746-marostegui.json
* 14:25 jgleeson: updated civicrm from {{Gerrit|0f195c6cca}} to {{Gerrit|d2e80f7522}}
* 14:21 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:20 marostegui: Upgrade mysql on db1091 after MCR changes
* 14:13 pt1979@cumin2001: START - Cookbook sre.dns.netbox
* 13:37 kormat@cumin1001: dbctl commit (dc=all): 'Repooling db1110 @ 100% [[phab:T261276|T261276]]', diff saved to https://phabricator.wikimedia.org/P12366 and previous config saved to /var/cache/conftool/dbconfig/20200826-133753-kormat.json
* 13:18 duesen: daniel@mwmaint1002:/srv/mediawiki/php-1.36.0-wmf.5$ mwscript maintenance/findBadBlobs.php dewiki --mark [[phab:T205936|T205936]] --revisions - < ~/T205936-dewiki-20050512070000.ids  # marking known bad revisions for [[phab:T205936|T205936]]
* 13:17 kormat@cumin1001: dbctl commit (dc=all): 'Repooling db1110 @ 75% [[phab:T261276|T261276]]', diff saved to https://phabricator.wikimedia.org/P12365 and previous config saved to /var/cache/conftool/dbconfig/20200826-131732-kormat.json
* 13:16 duesen: daniel@mwmaint1002:/srv/mediawiki/php-1.36.0-wmf.5$ mwscript maintenance/findBadBlobs.php oswiki --mark [[phab:T205936|T205936]] --revisions - < ~/T205936-oswiki-20090309200000.ids # marking known bad revisions for [[phab:T205936|T205936]]
* 13:07 kormat@cumin1001: dbctl commit (dc=all): 'Repooling db1110 @ 50% [[phab:T261276|T261276]]', diff saved to https://phabricator.wikimedia.org/P12364 and previous config saved to /var/cache/conftool/dbconfig/20200826-130735-kormat.json
* 13:06 vgutierrez: serve a synthetic warn page to DHE-RSA-AES128-SHA users - [[phab:T258405|T258405]]
* 12:47 kormat@cumin1001: dbctl commit (dc=all): 'Repooling db1110 @ 30% [[phab:T261276|T261276]]', diff saved to https://phabricator.wikimedia.org/P12363 and previous config saved to /var/cache/conftool/dbconfig/20200826-124700-kormat.json
* 12:21 kormat@cumin1001: dbctl commit (dc=all): 'Repooling db1110 @ 20% [[phab:T261276|T261276]]', diff saved to https://phabricator.wikimedia.org/P12362 and previous config saved to /var/cache/conftool/dbconfig/20200826-122059-kormat.json
* 12:12 godog: upgrade nagios-nrpe-server to 2.15-2 on jessie hosts - [[phab:T261198|T261198]]
* 11:58 kormat@cumin1001: dbctl commit (dc=all): 'Start repooling db1110 [[phab:T261276|T261276]]', diff saved to https://phabricator.wikimedia.org/P12361 and previous config saved to /var/cache/conftool/dbconfig/20200826-115850-kormat.json
* 11:56 mlitn@deploy1001: Synchronized php-1.36.0-wmf.6/extensions/WikibaseMediaInfo: MediaSearchQueryBuilder should support keyword only queries (duration: 01m 00s)
* 11:55 mlitn@deploy1001: Synchronized php-1.36.0-wmf.5/extensions/WikibaseMediaInfo: MediaSearchQueryBuilder should support keyword only queries (duration: 01m 08s)
* 11:53 kart_: Finished manual run of ContentTranslation/scripts/purge-unpublished-drafts.php script on mwmaint1002  ([[phab:T261189|T261189]])
* 11:39 kart_: Started manual run of ContentTranslation/scripts/purge-unpublished-drafts.php script on mwmaint1002  ([[phab:T261189|T261189]])
* 11:29 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/Wikibase.php: Config: [[gerrit:599344{{!}}Enable propagateChangeVisibility for testwikidata]], part 2 (duration: 01m 03s)
* 11:26 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:599344{{!}}Enable propagateChangeVisibility for testwikidata]], part 1 (duration: 01m 19s)
* 10:30 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 10:28 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 10:14 elukey@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 10:14 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 09:18 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 09:14 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 09:00 XioNoX: re-enable IPv6 BGP to Init7 in knams
* 08:40 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1110 replication broken', diff saved to https://phabricator.wikimedia.org/P12360 and previous config saved to /var/cache/conftool/dbconfig/20200826-084044-marostegui.json
* 08:16 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 08:14 elukey@cumin1001: START - Cookbook sre.hosts.downtime
* 05:45 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1091 for MCR change', diff saved to https://phabricator.wikimedia.org/P12358 and previous config saved to /var/cache/conftool/dbconfig/20200826-054557-marostegui.json
* 05:44 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1114, db1135 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12357 and previous config saved to /var/cache/conftool/dbconfig/20200826-054409-marostegui.json
* 05:33 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1114, db1135 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12356 and previous config saved to /var/cache/conftool/dbconfig/20200826-053345-marostegui.json
* 05:23 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1114, db1135 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12355 and previous config saved to /var/cache/conftool/dbconfig/20200826-052355-marostegui.json
* 05:08 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1114, db1135 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12354 and previous config saved to /var/cache/conftool/dbconfig/20200826-050849-marostegui.json
* 05:03 marostegui: Update db1135 and db1114 after MCR changes
 
== 2020-08-25 ==
* 21:51 mutante: xhgui1001/xhgui2001 - Unpacking xhgui (0.12.0-2-wmf1) over (0.9.0-1-wmf1) ([[phab:T260397|T260397]])
* 21:50 mutante: xhgui1001 - Unpacking xhgui (0.12.0-2-wmf1) over (0.9.0-1-wmf1) ...
* 21:46 mutante: importing xhgui 0.12.0-2-wmf1 to buster-wikimedia APT repo ([[phab:T260397|T260397]])
* 19:40 dcausse@deploy1001: Finished deploy [wikimedia/discovery/analytics@125cb6d]: test: Add wikidata ttl import (duration: 00m 54s)
* 19:39 dcausse@deploy1001: Started deploy [wikimedia/discovery/analytics@125cb6d]: test: Add wikidata ttl import
* 19:15 marxarelli: 1.36.0-wmf.6 promoted to group0 ([[phab:T257974|T257974]]). no new errors
* 19:09 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.6
* 19:05 moritzm: installing Java security updates on cloudelastic* hosts
* 19:02 moritzm: installing Java security updates on elastic* hosts
* 18:19 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload
* 17:58 dduvall@deploy1001: Finished scap: testwikis wikis to 1.36.0-wmf.6 (duration: 41m 58s)
* 17:30 dcausse@deploy1001: Finished deploy [wikimedia/discovery/analytics@bc2f7f1]: test: Add wikidata ttl import (duration: 01m 52s)
* 17:28 dcausse@deploy1001: Started deploy [wikimedia/discovery/analytics@bc2f7f1]: test: Add wikidata ttl import
* 17:17 dduvall@deploy1001: Started scap: testwikis wikis to 1.36.0-wmf.6
* 17:08 dduvall@deploy1001: Pruned MediaWiki: 1.36.0-wmf.4 (duration: 01m 40s)
* 17:01 dduvall@deploy1001: Pruned MediaWiki: 1.36.0-wmf.3 (duration: 19m 12s)
* 17:01 herron: imported logstash, elasticsearch, and kibana 7.9.0 -oss packages into buster-wikimedia thirdparty/elastic79
* 16:42 dcausse@deploy1001: Finished deploy [wikimedia/discovery/analytics@89b4f74]: test: Add wikidata ttl import (duration: 00m 49s)
* 16:41 dcausse@deploy1001: Started deploy [wikimedia/discovery/analytics@89b4f74]: test: Add wikidata ttl import
* 16:21 shdubsh: restart logstash on logstash1007 -- gc duration outlier
* 16:08 dcausse@deploy1001: Finished deploy [wikimedia/discovery/analytics@ae6dd8d]: test: Add wikidata ttl import (duration: 00m 54s)
* 16:07 dcausse@deploy1001: Started deploy [wikimedia/discovery/analytics@ae6dd8d]: test: Add wikidata ttl import
* 16:00 gehel: repool wdqs1005 - catched up on lag
* 15:47 elukey: restart mariadb@analytics_meta on db1108 to apply a replication filter (exclude superset_staging database from replication)
* 15:44 jgleeson: fundraising-tools updated from {{Gerrit|dcad0bfe75}} to {{Gerrit|3fe3a23114}}
* 15:41 dcausse@deploy1001: Finished deploy [wikimedia/discovery/analytics@cbf2f9d]: Add wikidata ttl import (duration: 01m 38s)
* 15:39 dcausse@deploy1001: Started deploy [wikimedia/discovery/analytics@cbf2f9d]: Add wikidata ttl import
* 15:22 liw: testing upcoming Scap release on beta
* 14:56 moritzm: installing rake security updates on stretch
* 14:56 moritzm: installing take security updates on stretch
* 14:36 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 14:32 volker-e@deploy1001: Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide:  (duration: 00m 05s)
* 14:32 volker-e@deploy1001: Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide:
* 14:26 XioNoX: disable IPv6 BGP to Init7 in knams
* 14:10 andrew@deploy1001: Finished deploy [horizon/deploy@7a3221d]: add hostname checking --bug [[phab:T207538|T207538]] (duration: 03m 50s)
* 14:06 andrew@deploy1001: Started deploy [horizon/deploy@7a3221d]: add hostname checking --bug [[phab:T207538|T207538]]
* 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1114 for MCR change', diff saved to https://phabricator.wikimedia.org/P12347 and previous config saved to /var/cache/conftool/dbconfig/20200825-135248-marostegui.json
* 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'fully repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12346 and previous config saved to /var/cache/conftool/dbconfig/20200825-134736-marostegui.json
* 13:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12345 and previous config saved to /var/cache/conftool/dbconfig/20200825-133734-marostegui.json
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12344 and previous config saved to /var/cache/conftool/dbconfig/20200825-132027-marostegui.json
* 13:17 moritzm: installing firejail security updates on remaining mw* servers in eqiad
* 12:56 godog: upgrade nagios-nrpe-server on scb2* and mwlog* - [[phab:T261198|T261198]]
* 12:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12343 and previous config saved to /var/cache/conftool/dbconfig/20200825-125108-marostegui.json
* 12:45 marostegui: Update MySQL on db1111 after MCR change
* 12:39 marostegui: alter table sites on s6, directly on the primary master [[phab:T260476|T260476]]
* 12:39 godog: test nagios-nrpe-server with dh 2048 on scb2001 - [[phab:T261198|T261198]]
* 12:35 moritzm: imported ceph packages from stretch-backports to component/ceph [[phab:T256877|T256877]]
* 12:10 moritzm: installing ruby-json security updates
* 12:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1135 MCR change', diff saved to https://phabricator.wikimedia.org/P12341 and previous config saved to /var/cache/conftool/dbconfig/20200825-120708-marostegui.json
* 12:02 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12340 and previous config saved to /var/cache/conftool/dbconfig/20200825-120211-marostegui.json
* 11:59 jayme@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' .
* 11:49 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12339 and previous config saved to /var/cache/conftool/dbconfig/20200825-114938-marostegui.json
* 11:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12338 and previous config saved to /var/cache/conftool/dbconfig/20200825-113758-marostegui.json
* 11:36 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 11:32 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12337 and previous config saved to /var/cache/conftool/dbconfig/20200825-112859-marostegui.json
* 11:25 marostegui: Upgrade mysql on db1118 after MCR change
* 11:16 Urbanecm: EU B&C done
* 11:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|d869e308492ee72cb3d1998b15409aa44a4af9c7}}: Enable ContentTranslation as a default tool in Assamese and Burmese WPs ([[phab:T258503|T258503]]; [[phab:T258505|T258505]]) (duration: 01m 00s)
* 10:59 moritzm: installing remaining libx11 security updates
* 10:37 arturo: import all binary packages from tesseract-ocr-lang into stretch-wikimedia/component/tesseract-410-bpo ([[phab:T247422|T247422]])
* 10:32 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 10:28 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 10:28 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 10:23 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 10:23 moritzm: removed fermium.wikimedia.org from debmonitor
* 09:45 marostegui: Create missing table cx_notification_log on x1 wikishared [[phab:T261190|T261190]]
* 08:50 XioNoX: re-activate eqord peering/transit - [[phab:T259593|T259593]]
* 08:19 XioNoX: reconfigure eqord to be AS65020 - [[phab:T259593|T259593]]
* 08:18 XioNoX: deactivate eqord peering/transit - [[phab:T259593|T259593]]
* 07:22 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=1)
* 07:13 marostegui: Upgrade MySQL on dbstore1004
* 07:09 dcausse: depooling wdqs1005 (high lag)
* 07:04 dcausse: restartint blazegraph on wdqs1005 ([[phab:T242453|T242453]])
* 06:20 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 05:38 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1111, db1118 for MCR change', diff saved to https://phabricator.wikimedia.org/P12336 and previous config saved to /var/cache/conftool/dbconfig/20200825-053856-marostegui.json
* 05:38 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12335 and previous config saved to /var/cache/conftool/dbconfig/20200825-053801-marostegui.json
* 05:26 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12334 and previous config saved to /var/cache/conftool/dbconfig/20200825-052602-marostegui.json
* 05:21 moritzm: installing Java security updates on relforge*
* 05:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12333 and previous config saved to /var/cache/conftool/dbconfig/20200825-051327-marostegui.json
* 05:11 marostegui: Remove revisions triggers from db2094:3311 [[phab:T238966|T238966]]
* 05:10 marostegui: Deploy MCR schema change on s1 codfw, this will create lag on s1 codfw - [[phab:T238966|T238966]]
* 05:04 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12332 and previous config saved to /var/cache/conftool/dbconfig/20200825-050451-marostegui.json
* 04:02 ejegg: updated fundraising python tools from {{Gerrit|305f2a4438}} to {{Gerrit|dcad0bfe75}}
* 01:49 eileen: civicrm revision changed from {{Gerrit|ce28723709}} to {{Gerrit|0f195c6cca}}, config revision is {{Gerrit|96839009f1}}
* 01:39 eileen: civicrm revision is {{Gerrit|ce28723709}}, config revision is {{Gerrit|96839009f1}}
* 01:30 eileen: civicrm revision is {{Gerrit|ce28723709}}, config revision is {{Gerrit|54c8c7abf2}}
* 01:17 cdanis: repool esams
* 01:11 cdanis: [[phab:T259621|T259621]] wrong junos version was staged on cr2-esams, abandoning this attempt and putting back in service
* 01:07 cdanis: cdanis@re0.cr2-esams> request system software add validate re1 /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz
* 00:56 cdanis: [[phab:T259621|T259621]] ❌cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 homer 'cr*' commit 'drain cr2-esams transport link'
* 00:36 cdanis: [[phab:T259621|T259621]] cdanis@re1.cr3-esams> request chassis routing-engine master switch
* 00:30 cdanis: [[phab:T259621|T259621]] cdanis@re1.cr3-esams> request vmhost reboot re0
* 00:24 cdanis: [[phab:T259621|T259621]] cdanis@re1.cr3-esams> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz re0
* 00:18 cdanis: [[phab:T259621|T259621]] cdanis@re0.cr3-esams> request chassis routing-engine master switch
* 00:14 cdanis: [[phab:T259621|T259621]] cdanis@re0.cr3-esams> request vmhost reboot re1
* 00:08 cdanis: [[phab:T259621|T259621]] cdanis@re0.cr3-esams> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz re1
 
== 2020-08-24 ==
* 23:46 cdanis: depool esams [[phab:T259621|T259621]]
* 23:16 Urbanecm: Evening B&C window done
* 23:06 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|778f710bbbdb24730f7ce4c75d5ff1ca7a5ce3b3}}: Alternate configuration mechanism for Parsoid ([[phab:T241961|T241961]]) (duration: 00m 58s)
* 22:13 rzl@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
* 22:10 rzl@cumin1001: START - Cookbook sre.hosts.decommission
* 21:29 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Deployed additional mitigations for [[phab:T257687|T257687]] (duration: 00m 58s)
* 20:29 rzl: re-enabled puppet on 'R:File = /etc/nutcracker/nutcracker.yml' [[phab:T261154|T261154]]
* 19:25 rzl: disabling puppet on 'R:File = /etc/nutcracker/nutcracker.yml' to swap mc2028 out for mc2037 [[phab:T261154|T261154]]
* 18:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: cirrus: Increase weight of grants and research namespaces in metawiki search (duration: 00m 58s)
* 15:20 jynus: shutdown backup2001 [[phab:T260764|T260764]]
* 15:13 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 15:08 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 15:04 vgutierrez: rolling restart of ats-tls to disable ECDHE-RSA-AES128-SHA - [[phab:T258405|T258405]]
* 14:58 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 14:55 rzl: switchover test complete, puppet re-enabled on cumin1001
* 14:54 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0)
* 14:53 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-update-tendril
* 14:53 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0)
* 14:52 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance
* 14:52 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0)
* 14:52 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 14:48 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99)
* 14:48 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 14:47 godog: powercycle ganeti5002 -- host down and nothing in console
* 14:43 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0)
* 14:43 rzl@cumin1001: [DRY-RUN] MediaWiki read-only period ends at: 2020-08-24 14:43:35.570234
* 14:43 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.07-set-readwrite
* 14:43 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)
* 14:43 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
* 14:43 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0)
* 14:43 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions
* 14:43 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0)
* 14:43 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki
* 14:42 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=99)
* 14:42 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
* 14:42 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0)
* 14:41 rzl@cumin1001: [DRY-RUN] MediaWiki read-only period starts at: 2020-08-24 14:41:55.754938
* 14:41 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.02-set-readonly
* 14:41 dcausse: creating cirrus indices for lldwiki
* 14:39 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0)
* 14:39 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance
* 14:38 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0)
* 14:28 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-warmup-caches
* 14:28 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0)
* 14:28 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 14:24 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99)
* 14:24 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 14:24 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0)
* 14:24 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-disable-puppet
* 14:22 moritzm: installing libexif security updates on stretch
* 14:18 rzl: disabling puppet on cumin1001 and starting a test of the DC switchover automation, expect some SAL noise but no production impact
* 14:08 duesen: Deployed patch for [[phab:T260485|T260485]]
* 13:59 marostegui: Stop mysql on db1117:3325 to clone db1128 - [[phab:T260324|T260324]]
* 13:55 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1092 for MCR change', diff saved to https://phabricator.wikimedia.org/P12327 and previous config saved to /var/cache/conftool/dbconfig/20200824-135538-marostegui.json
* 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1101:3318 after MCR change', diff saved to https://phabricator.wikimedia.org/P12326 and previous config saved to /var/cache/conftool/dbconfig/20200824-133032-marostegui.json
* 13:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 13:25 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 13:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P12325 and previous config saved to /var/cache/conftool/dbconfig/20200824-131305-marostegui.json
* 13:05 moritzm: installing imagemagick security updates on stretch
* 13:00 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P12323 and previous config saved to /var/cache/conftool/dbconfig/20200824-130024-marostegui.json
* 12:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P12322 and previous config saved to /var/cache/conftool/dbconfig/20200824-125131-marostegui.json
* 12:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1084 for MCR change', diff saved to https://phabricator.wikimedia.org/P12321 and previous config saved to /var/cache/conftool/dbconfig/20200824-122848-marostegui.json
* 12:27 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1105:3311 after MCR change', diff saved to https://phabricator.wikimedia.org/P12320 and previous config saved to /var/cache/conftool/dbconfig/20200824-122752-marostegui.json
* 12:20 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12319 and previous config saved to /var/cache/conftool/dbconfig/20200824-122050-marostegui.json
* 12:12 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12318 and previous config saved to /var/cache/conftool/dbconfig/20200824-121200-marostegui.json
* 12:03 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1105:3311', diff saved to https://phabricator.wikimedia.org/P12317 and previous config saved to /var/cache/conftool/dbconfig/20200824-120310-marostegui.json
* 12:01 Urbanecm: EU B&C window completed
* 12:01 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|8c380d65d760591099c296ae522b2e63953413aa}}: Enable tewiki as import source for tewikibooks ([[phab:T260107|T260107]]) (duration: 00m 57s)
* 11:58 XioNoX: test advertise CF tunnel endpoint on cr1-eqiad - [[phab:T259036|T259036]]
* 11:57 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|5a6d025b04eb20787e8abbbdd56a3abb3818b82f}}: Add retrobibliothek.de to the wgCopyUploadsDomains allowlist of Wikimedia Commons ([[phab:T261012|T261012]]) (duration: 00m 56s)
* 11:50 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|e1ae39afbb4d6f33e74782580db7dfee06d0097d}}: Enable mapframe at trwiki ([[phab:T260594|T260594]]) (duration: 00m 58s)
* 11:43 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: {{Gerrit|1066ecbe2836e69211c905f597ad6b62241528c0}}: Enable MediaSearch A/B test ([[phab:T254388|T254388]]) (duration: 00m 56s)
* 11:42 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.5/extensions/ContentTranslation/modules/publish/ext.cx.wikibase.link.js: {{Gerrit|74a87184408937bcdb4a27f1f563bbbdff45cf97}}: Publish: Fix broken wikidata linking ([[phab:T249458|T249458]]) (duration: 00m 58s)
* 11:39 Urbanecm: Purge 13 URLs with purgeList.php, see P12316 for list of them ([[phab:T260908|T260908]]; [[phab:T258552|T258552]]; [[phab:T261076|T261076]]; [[phab:T261110|T261110]])
* 11:34 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 11:32 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 11:32 arturo: add liblept5 1.76.0-1~bpo9+1 (and leptonica-progs) to stretch-wikimedia/component/tesseract-410-bpo ([[phab:T247422|T247422]])
* 11:30 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|fe0449d244ee876e4fb64da630f0994ab114f248}}: {{Gerrit|74220d0943e6b32cce3c93dd5b9f8bbc63fa5d73}}: {{Gerrit|7db8a19c512cea84f3000463e9dfb6617857c9a6}}: Update Chinese wordmarks and taglines, update zhwikisource project logo ([[phab:T260908|T260908]]; [[phab:T258552|T258552]]; [[phab:T261076|T261076]]; [[phab:T261110|T261110]]) (duration: 00m 59s)
* 11:29 urbanecm@deploy1001: Synchronized static/images/: {{Gerrit|fe0449d244ee876e4fb64da630f0994ab114f248}}: {{Gerrit|74220d0943e6b32cce3c93dd5b9f8bbc63fa5d73}}: {{Gerrit|7db8a19c512cea84f3000463e9dfb6617857c9a6}}: Update Chinese wordmarks and taglines, update zhwikisource project logo ([[phab:T260908|T260908]]; [[phab:T258552|T258552]]; [[phab:T261076|T261076]]; [[phab:T261110|T261110]]) (duration: 00m 58s)
* 11:21 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 10:46 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:622116{{!}} Bumping portals to master (T128546)]] (duration: 00m 58s)
* 10:45 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:622116{{!}} Bumping portals to master (T128546)]] (duration: 01m 00s)
* 10:43 moritzm: installing ruby2.3 security updates
* 10:12 moritzm: installing firejail security updates on mw canaries
* 09:58 oblivian@cumin1001: conftool action : set/weight=1; selector: dc=codfw,cluster=appserver,service=canary
* 09:46 XioNoX: add PNI to CF on cr1-eqiad with import/export NONE - [[phab:T259036|T259036]]
* 09:18 moritzm: restarting mw canaries to pick up libx11 update
* 09:13 moritzm: installing libx11 security updates on stretch
* 09:10 vgutierrez: repool cp5002
* 09:08 _joe_: restarting php-fpm on mw1344 (stuck in SIGILL for new children)
* 09:00 vgutierrez: restart ats-tls on cp5002
* 08:54 moritzm: installing net-snmp security updates on buster
* 08:52 ema: depool cp5002 due to icinga errors
* 08:24 moritzm: installing json-c security updates on buster
* 07:36 XioNoX: push new pfw policies - [[phab:T261007|T261007]]
* 05:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1101:3318, db1105:3311 for MCR change', diff saved to https://phabricator.wikimedia.org/P12315 and previous config saved to /var/cache/conftool/dbconfig/20200824-052916-marostegui.json
 
== 2020-08-23 ==
* 20:38 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 20:36 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
* 20:18 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 20:15 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
* 20:02 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 20:00 cmjohnson@cumin1001: START - Cookbook sre.hosts.downtime
* 11:23 gehel: repool wdqs1006 - catched up on lag
 
== 2020-08-22 ==
* 19:33 ryankemper: depooled wdqs1006 (still has 2.5 hours to catch up on)
* 19:31 ryankemper: pooled wdqs1006 now that lag has dissipated
* 07:36 gehel: restart blazegraph on wdqs1006 + depool to catchup on lag
* 05:24 legoktm: legoktm@mwmaint1002:~$ echo "https://releases.wikimedia.org/mediawiki/1.35/" {{!}} mwscript purgeList.php --wiki=aawiki
 
== 2020-08-21 ==
* 17:43 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:39 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:00 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 16:58 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 16:41 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 16:39 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 16:17 zpapierski@deploy1001: Finished deploy [search/mjolnir/deploy@c80e2e7]: .. redeploy after theory verification (duration: 00m 50s)
* 16:16 zpapierski@deploy1001: Started deploy [search/mjolnir/deploy@c80e2e7]: .. redeploy after theory verification
* 16:15 zpapierski@deploy1001: deploy aborted: .. (duration: 00m 01s)
* 16:15 zpapierski@deploy1001: Started deploy [search/mjolnir/deploy@c80e2e7]: ..
* 13:25 jayme@cumin1001: conftool action : set/pooled=True; selector: dnsdisc=termbox,name=codfw
* 13:25 jayme@cumin1001: conftool action : set/pooled=False; selector: dnsdisc=termbox,name=codfw
* 09:56 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 09:55 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 09:02 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 09:01 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 01:51 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 01:49 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 01:15 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 01:13 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
 
== 2020-08-20 ==
* 22:31 eileen: civicrm revision changed from {{Gerrit|27d5900f7d}} to {{Gerrit|ce28723709}}, config revision is {{Gerrit|706cf3c898}}
* 22:20 eileen: civicrm revision is {{Gerrit|27d5900f7d}}, config revision is {{Gerrit|706cf3c898}}
* 22:20 mutante: permanently shut down tungsten.eqiad.wmnet [[phab:T260395|T260395]] [[phab:T158837|T158837]] [[phab:T180761|T180761]] [[phab:T224549|T224549]]
* 22:18 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 22:17 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 21:35 ejegg: updated fundraising CiviCRM from {{Gerrit|958a79f660}} to {{Gerrit|27d5900f7d}}
* 20:53 cdanis: repool eqsin
* 20:37 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 20:36 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 20:34 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 20:25 cdanis: cdanis@cr2-eqsin> request vmhost reboot
* 20:17 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 20:16 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 20:13 cdanis: cdanis@cr2-eqsin> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-18.2R3-S5.3.tgz
* 20:11 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 20:02 cdanis: depool eqsin for router upgrade
* 19:57 ayounsi@cumin1001: END (ERROR) - Cookbook sre.network.prepare-upgrade (exit_code=97)
* 19:37 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 19:34 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 19:34 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 19:24 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 19:17 bstorm@cumin1001: END (FAIL) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=99)
* 19:17 bstorm@cumin1001: START - Cookbook wmcs.wikireplicas.add_wiki
* 19:17 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.5  refs [[phab:T257973|T257973]]
* 19:08 mutante: restarted apache on cont2001 for integration.wikimedia.org docroot change
* 19:07 mutante: switching document root of integration.wikimedia.org to scap ([[phab:T149924|T149924]])
* 19:02 twentyafterfour: 1.36.0-wmf.5 has no known blockers and logspam is cleaned up, time to roll group2 wikis to wmf.5
* 18:42 bstorm@cumin1001: END (FAIL) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=99)
* 18:42 bstorm@cumin1001: START - Cookbook wmcs.wikireplicas.add_wiki
* 18:19 mutante: ores1004 - starting failed celery-ores-worker
* 18:18 mutante: testreduce1001 - rt_client and vd_client now properly stopped by puppet [[phab:T257906|T257906]]
* 17:29 shdubsh: restart elasticsearch on logstash1012 (not 1020) -- high gc runtimes
* 17:28 shdubsh: restart elasticsearch on logstash1020 -- high gc runtimes
* 17:23 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 17:23 ayounsi@cumin1001: END (ERROR) - Cookbook sre.network.prepare-upgrade (exit_code=97)
* 17:23 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 17:22 ayounsi@cumin1001: END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99)
* 17:22 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 16:48 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:48 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:46 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:45 pt1979@cumin2001: START - Cookbook sre.dns.netbox
* 16:43 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:40 _joe_: restarted apache2 on icinga1001
* 16:13 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:11 shdubsh: restart elasticsearch on logstash1011 -- long gc runs
* 16:10 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:08 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:02 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 14:55 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 14:06 oblivian@deploy1001: Finished deploy [ores/deploy@8540eec]: various configuration fixes (duration: 09m 03s)
* 13:57 oblivian@deploy1001: Started deploy [ores/deploy@8540eec]: various configuration fixes
* 13:53 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 13:53 oblivian@deploy1001: Finished deploy [ores/deploy@e860508]: switch everything to use envoy as a service proxy [[phab:T244843|T244843]] (duration: 14m 00s)
* 13:39 oblivian@deploy1001: Started deploy [ores/deploy@e860508]: switch everything to use envoy as a service proxy [[phab:T244843|T244843]]
* 13:26 oblivian@deploy1001: Finished deploy [ores/deploy@74677b6]: switch testwiki to use envoy as a service proxy [[phab:T244843|T244843]] (take 2) (duration: 11m 37s)
* 13:14 oblivian@deploy1001: Started deploy [ores/deploy@74677b6]: switch testwiki to use envoy as a service proxy [[phab:T244843|T244843]] (take 2)
* 13:11 oblivian@deploy1001: Finished deploy [ores/deploy@a208a0e]: switch testwiki to use envoy as a service proxy [[phab:T244843|T244843]] (duration: 11m 19s)
* 13:09 gehel: repool wdqs1007 - catched up on lag
* 13:00 oblivian@deploy1001: Started deploy [ores/deploy@a208a0e]: switch testwiki to use envoy as a service proxy [[phab:T244843|T244843]]
* 12:51 oblivian@deploy1001: Finished deploy [ores/deploy@a208a0e]: switch testwiki to use envoy as a service proxy [[phab:T244843|T244843]] (duration: 07m 03s)
* 12:44 oblivian@deploy1001: Started deploy [ores/deploy@a208a0e]: switch testwiki to use envoy as a service proxy [[phab:T244843|T244843]]
* 11:49 Lucas_WMDE: EU backport window done
* 11:44 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.4/extensions/AbuseFilter/includes/AbuseFilterHooks.php: {{Gerrit|d762e7b5526d91fe21e5980bc5e9f3be06a2f85c}}: Use $user param when filtering edits ([[phab:T258717|T258717]]) (duration: 01m 05s)
* 11:41 eileen: civicrm revision changed from {{Gerrit|6c9441a18e}} to {{Gerrit|958a79f660}}, config revision is {{Gerrit|706cf3c898}}
* 11:38 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.5/extensions/AbuseFilter/includes/AbuseFilterHooks.php: {{Gerrit|00da39b6913ac2eab600bbb61258472b60d2cbcb}}: Use $user param when filtering edits ([[phab:T258717|T258717]]) (duration: 01m 05s)
* 11:32 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.36.0-wmf.5/extensions/Wikibase/client/data-bridge/dist/: Backport: [[gerrit:621488{{!}}Don't try to load source maps in production (T260852)]] (duration: 01m 07s)
* 11:07 mlitn@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix testwikidata depicts id & CirrusSearchUserTesting config (duration: 01m 06s)
* 11:07 Urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=trwiki editor # [[phab:T260899|T260899]]
* 10:58 XioNoX: re-pool codfw - [[phab:T259621|T259621]]
* 10:53 XioNoX: un-drain cr1-codfw - [[phab:T259621|T259621]]
* 10:45 XioNoX: cr1-codfw> request chassis routing-engine master switch - [[phab:T259621|T259621]]
* 10:26 hashar: Restarted zuul-merger instances on contint1001 and contint2001
* 10:24 hashar@deploy1001: Finished deploy [zuul/deploy@8a05b4d]: Support Gerrit replication events (duration: 00m 24s)
* 10:24 hashar@deploy1001: Started deploy [zuul/deploy@8a05b4d]: Support Gerrit replication events
* 10:21 XioNoX: cr1-codfw> request chassis routing-engine master switch - [[phab:T259621|T259621]]
* 10:12 XioNoX: reboot cr1-codfw:re1 (backup) for upgrade - [[phab:T259621|T259621]]
* 09:57 XioNoX: bump cr1-codfw OSPF metrics - [[phab:T259621|T259621]]
* 09:51 XioNoX: enable transit/peering and re-set normal OSPF values on cr2-codfw - [[phab:T259621|T259621]]
* 09:41 XioNoX: cr2-codfw> request chassis routing-engine master switch - [[phab:T259621|T259621]]
* 09:36 eileen: civicrm revision changed from {{Gerrit|cf9fadbeed}} to {{Gerrit|6c9441a18e}}, config revision is {{Gerrit|706cf3c898}}
* 09:33 XioNoX: reboot cr2-codfw:re0 (backup) for upgrade - [[phab:T259621|T259621]]
* 09:18 XioNoX: cr2-codfw> request chassis routing-engine master switch - [[phab:T259621|T259621]]
* 09:18 kormat: stress-testing db2125 [[phab:T260670|T260670]]
* 09:08 XioNoX: reboot cr2-codfw:re1 (backup) for upgrade - [[phab:T259621|T259621]]
* 09:03 kormat@cumin1001: dbctl commit (dc=all): 'Repool db2125 after host failure [[phab:T260670|T260670]]', diff saved to https://phabricator.wikimedia.org/P12303 and previous config saved to /var/cache/conftool/dbconfig/20200820-090313-kormat.json
* 08:52 kormat: removing /usr/bin/check_mariadb.py from all db hosts [[phab:T259516|T259516]]
* 08:52 XioNoX: disable transit/peering on cr2-codfw - [[phab:T259621|T259621]]
* 08:48 XioNoX: bump cr2-codfw OSPF metrics - [[phab:T259621|T259621]]
* 08:44 jynus: running analyze table on db1115's tendril.global_status_log, may case some stalls on tendril/dbtree [[phab:T260876|T260876]]
* 08:41 XioNoX: depool codfw for routers upgrade - [[phab:T259621|T259621]]
* 08:31 XioNoX: enable transit/peering on cr3-knams - [[phab:T259621|T259621]]
* 08:21 XioNoX: reboot cr3-knams for upgrade - [[phab:T259621|T259621]]
* 08:07 XioNoX: disable transit/peering on cr3-knams - [[phab:T259621|T259621]]
* 07:39 hashar: contint2001: restarted zuul
* 07:29 hashar: contint1001: restarted zuul-merger
* 07:29 hashar@deploy1001: Finished deploy [zuul/deploy@5989ed0]: Upgrade gear from 0.7.0 to 1.15.1+wmf1 - [[phab:T258630|T258630]] (duration: 00m 13s)
* 07:28 hashar@deploy1001: Started deploy [zuul/deploy@5989ed0]: Upgrade gear from 0.7.0 to 1.15.1+wmf1 - [[phab:T258630|T258630]]
* 01:54 ejegg: re-enabled fundraising scheduled jobs
* 00:51 mutante: ms-be1039 - started failed ferm service
* 00:35 ejegg: stopped fundraising scheduled jobs
* 00:27 eileen: civicrm revision changed from {{Gerrit|c442a09153}} to {{Gerrit|cf9fadbeed}}, config revision is {{Gerrit|3cdffd4fc2}}
 
== 2020-08-19 ==
* 23:20 Urbanecm: Evening B&C window closed
* 23:08 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|a80899948c26ca36b970b80fbad07600fe4ce92c}}: Enable VisualEditor in namespaces Draft and Wikiproject on hywiki ([[phab:T260825|T260825]]) (duration: 01m 05s)
* 22:41 eileen: civicrm revision changed from {{Gerrit|34f95a3311}} to {{Gerrit|c442a09153}}, config revision is {{Gerrit|3cdffd4fc2}}
* 21:27 eileen: civicrm revision changed from {{Gerrit|154519cc1f}} to {{Gerrit|34f95a3311}}, config revision is {{Gerrit|3cdffd4fc2}}
* 21:17 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
* 21:17 cdanis@cumin1001: START - Cookbook sre.network.cf
* 20:39 dpifke@deploy1001: Finished deploy [performance/arc-lamp@2ef1af7]: Deploy fixes for notifications and OOM prevention ([[phab:T259167|T259167]]) (duration: 00m 06s)
* 20:39 dpifke@deploy1001: Started deploy [performance/arc-lamp@2ef1af7]: Deploy fixes for notifications and OOM prevention ([[phab:T259167|T259167]])
* 19:43 ebernhardson: restart mjolnir-kafka-bulk-daemon on search-loader2001 with debug logging
* 19:20 mutante: testreduce1001 - re-enabled puppet, confirmed parsoid-rt service was now stopped properly by puppet while it runs as before on scandium, the previous parsoid-testing host. switching it over is now a Hiera one-liner. ([[phab:T257906|T257906]])
* 19:15 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.5  refs [[phab:T257973|T257973]] (duration: 01m 04s)
* 19:14 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.5  refs [[phab:T257973|T257973]]
* 19:02 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|60af096b80a8ef7bc94ec40ce203fd27b0c97f26}}: Add autopatrolled group at arzwiki ([[phab:T260761|T260761]]) (duration: 01m 04s)
* 18:52 mutante: testreduce1001 - disable puppet; stop parsoid-rt service
* 18:47 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|924a03bd624d6750a7e776e09713056cc45e5cc5}}: Add clinton.presidentiallibraries.us to the wgCopyUploadsDomains allowlist of Wikimedia Commons ([[phab:T259927|T259927]]) (duration: 01m 04s)
* 18:45 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|83b34e1bd1ed804a70f67e089580e082f89e2a0f}}: ClosedWikiProvider: Use testUserForCreation rather than testForAuthentication ([[phab:T258695|T258695]]) (duration: 01m 04s)
* 18:39 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|95d45f6e002df78d4860a711042d77a6b0bdecb9}}: Dont index Draft (118) and Draft talk (119) on hywiki ([[phab:T260804|T260804]]) (duration: 01m 04s)
* 18:32 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|803cb1a0d2c8cc6df8e4e88ab3c4d27eb71d01b3}}: Update taglines for various projects ([[phab:T258552|T258552]]) (duration: 01m 04s)
* 18:30 urbanecm@deploy1001: Synchronized static/images/mobile/copyright/: {{Gerrit|803cb1a0d2c8cc6df8e4e88ab3c4d27eb71d01b3}}: Update taglines for various projects ([[phab:T258552|T258552]]) (duration: 01m 06s)
* 18:25 mutante: rebooting webperf1002 VM on ganeti level (outside OS) to upgrade rom 8 to 16GB RAM ([[phab:T260192|T260192]])
* 18:24 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 18:24 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 18:24 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|bb4aa44b0bd5b2b33d190d3af81e038e5fc55e3f}}: Configure namespaces on commons to include categories ([[phab:T198716|T198716]]) (duration: 01m 04s)
* 18:21 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|b9043331c1c1b352256cffd471b9ff128806607c}}: Update project wordmarks ([[phab:T254788|T254788]]; sync 2/2) (duration: 01m 04s)
* 18:19 urbanecm@deploy1001: Synchronized static/images/mobile/copyright/: {{Gerrit|b9043331c1c1b352256cffd471b9ff128806607c}}: Update project wordmarks ([[phab:T254788|T254788]]; sync 1/2) (duration: 01m 06s)
* 18:15 mutante: rebooting webperf2002 VM on ganeti level (outside OS) to upgrade rom 8 to 16GB RAM ([[phab:T260192|T260192]])
* 18:15 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|a6f8354e7599a5e92bea060807065f5b42c540e5}}: Enable $wgMFNoindexPages for all wikis ([[phab:T255458|T255458]]) (duration: 01m 07s)
* 18:13 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 18:13 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 18:13 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:38 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
* 17:38 mutante: decom'ing releases2001.codfw.wmnet (
* 17:37 dzahn@cumin1001: START - Cookbook sre.hosts.decommission
* 16:39 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:37 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:32 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 16:30 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:41 rzl: finished exercising the switchdc cookbooks with --live-test for now, all changes reverted including re-enabling puppet on cumin1001
* 15:38 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0)
* 15:37 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance
* 15:34 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0)
* 15:34 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 15:33 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99)
* 15:33 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 15:31 jbond42: update java.security https://gerrit.wikimedia.org/r/c/operations/puppet/+/593467
* 15:30 oblivian@cumin1001: conftool action : set/ttl=300; selector: dnsdisc=api-rw
* 15:26 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99)
* 15:26 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 15:22 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99)
* 15:22 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 15:18 godog: prometheus codfw lvextend --resizefs --size +80G /dev/mapper/vg--ssd-prometheus--ops
* 15:17 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0)
* 15:17 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 15:16 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99)
* 15:16 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 15:14 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0)
* 15:14 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 15:08 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99)
* 15:08 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.08-restore-ttl
* 15:06 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:04 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 14:50 rzl@cumin1001: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99)
* 14:50 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl
* 14:50 rzl: running the switchdc cookbooks with --live-test, simulating a switch to eqiad where we're already running, no production impact is expected
* 14:47 rzl@cumin1001: END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0)
* 14:47 rzl@cumin1001: START - Cookbook sre.switchdc.mediawiki.00-disable-puppet
* 14:41 rzl: disable puppet on cumin1001 for switchdc testing
* 14:35 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 14:33 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 14:27 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 13:38 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 13:34 gehel: depooling wdqs1007 and restarting blazegraph
* 13:29 _joe_: depooling and disabling puppet on restbase1024 for further investigation
* 13:27 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 13:26 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 13:25 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 13:10 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:03 _joe_: building and uploading fluent-bit, ratelimit images
* 13:01 pt1979@cumin2001: START - Cookbook sre.dns.netbox
* 12:57 _joe_: building a new version of the base docker images
* 11:29 awight: EU bacon finished
* 11:28 effie: restart mwdebug* servers
* 11:08 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:621227{{!}}Fix typos in flaggedrevs comments ()]] (duration: 01m 19s)
* 09:22 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 08:48 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 08:48 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 08:43 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 08:43 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 08:36 XioNoX: update firewall policies on pfw - [[phab:T260585|T260585]]
* 08:35 jayme: running puppet on A:all-mw-eqiad
* 08:23 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 08:23 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 08:20 godog: switch grafana.w.o to grafana 7 in codfw - [[phab:T259143|T259143]]
* 08:19 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 08:18 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 08:14 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 08:06 jayme: running puppet on A:all-mw-eqiad
* 07:46 godog: upgrade to grafana 7 on cloudmetrics hosts - [[phab:T259143|T259143]]
* 07:15 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 07:10 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 06:39 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 06:13 eileen: tools revision changed from {{Gerrit|b4ebd1e564}} to {{Gerrit|0b9d971bc4}}
* 06:07 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 06:04 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 06:03 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 06:00 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 05:55 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 05:53 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 05:47 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 05:37 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 05:31 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 03:42 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 03:40 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 02:53 cstone: civicrm revision changed from {{Gerrit|f5469d0a4c}} to {{Gerrit|154519cc1f}}
* 02:00 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 01:05 dpifke@deploy1001: Synchronized wmf-config/profiler.php: Deploying https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620139 (duration: 01m 18s)
* 00:49 dpifke@deploy1001: Synchronized wmf-config/ProductionServices.php: Disabling old XHGui backend ([[phab:T180761|T180761]]) (duration: 05m 13s)
* 00:15 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-cluster
 
== 2020-08-18 ==
* 23:45 catrope@deploy1001: Synchronized php-1.36.0-wmf.5/extensions/GrowthExperiments: Only fetch task card data for users in variant C/D ([[phab:T258021|T258021]]) (duration: 01m 05s)
* 23:44 catrope@deploy1001: Synchronized php-1.36.0-wmf.4/extensions/GrowthExperiments: Only fetch task card data for users in variant C/D ([[phab:T258021|T258021]]) (duration: 01m 06s)
* 23:38 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1301.eqiad.wmnet
* 23:34 Urbanecm: Run scap pull at mw1301
* 23:33 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable static maps on testwiki, disable them on test2wiki (duration: 03m 22s)
* 23:32 mutante: rebooting mw1301 via mgmt
* 23:22 mutante: killed reboot-cluster on cumin1001
* 23:09 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|ac34f7274823e40d0c79752eb5ffe74c76856d04}}: Enable subpages in NS:0 in techconductwiki ([[phab:T260350|T260350]]) (duration: 05m 14s)
* 23:04 wkandek@cumin1001: conftool action : set/pooled=yes; selector: name=mw1300.eqiad.wmnet
* 22:47 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 22:41 wkandek@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 22:09 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
* 22:07 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
* 22:06 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
* 21:39 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
* 21:37 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
* 21:34 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
* 21:24 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 21:03 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 21:01 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 20:27 hashar: https://releases-jenkins.wikimedia.org/ changed agent from releases1001 to releases1002
* 20:14 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.5  refs [[phab:T257973|T257973]]
* 20:11 mutante: running puppet on cp-ats-ulsfo and switching releases-jenkins backend
* 20:07 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.36.0-wmf.5  refs [[phab:T257973|T257973]] (duration: 53m 12s)
* 20:00 mutante: releases1001 rm /etc/rsync.d/frag* & run puppet
* 19:54 mutante: rsyncing /var/lib/jenkins from releases1001 to releases1002/2002 with --delete  [[phab:T256164|T256164]]
* 19:47 ejegg: updated payments-wiki from {{Gerrit|a7ee1790e0}} to {{Gerrit|ef7ebd08cb}}
* 19:44 hashar: Deleting old jobs from https://releases-jenkins.wikimedia.org/  # [[phab:T256164|T256164]]
* 19:41 hashar: releases1001: deleting old legacy mediawiki snapshots under /var/lib/jenkins/<nowiki>{</nowiki>REL1_27,REL1_29,REL1_30<nowiki>}</nowiki>  # [[phab:T256164|T256164]]
* 19:14 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.36.0-wmf.5  refs [[phab:T257973|T257973]]
* 19:13 twentyafterfour: Promote testwikis from 1.36.0-wmf.4 to 1.36.0-wmf.5 refs [[phab:T257973|T257973]]
* 17:51 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 16:12 oblivian@cumin1001: conftool action : set/pooled=yes; selector: name=mw14(09{{!}}11{{!}}13).*
* 16:03 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 15:36 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 15:30 jayme@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 15:02 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 14:56 papaul: replacing msw-c1,c2 and c4
* 14:55 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 14:53 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P12293 and previous config saved to /var/cache/conftool/dbconfig/20200818-145337-marostegui.json
* 14:48 oblivian@cumin1001: conftool action : set/pooled=yes; selector: name=mw13(55{{!}}64{{!}}65).*
* 14:46 XioNoX: move v4 HE on cr3-ulsfo from peering to transit bgp group
* 14:44 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12292 and previous config saved to /var/cache/conftool/dbconfig/20200818-144415-marostegui.json
* 14:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12291 and previous config saved to /var/cache/conftool/dbconfig/20200818-143758-marostegui.json
* 14:35 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 14:29 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12290 and previous config saved to /var/cache/conftool/dbconfig/20200818-142937-marostegui.json
* 14:28 marostegui: Stop MYSQL on db2125 for on-site maintenance - [[phab:T260670|T260670]]
* 13:54 marostegui: Revoke DELETE and CREATE from xhgui user on m2 [[phab:T260640|T260640]]
* 13:53 XioNoX: bump Zayo v4 BGP session in eqiad
* 13:49 XioNoX: move v4 HE on cr2-eqord from peering to transit bgp group
* 13:37 XioNoX: move v4 cr1-eqiad from peering to transit bgp group
* 13:04 kormat: disabling puppet on all db machines [[phab:T259516|T259516]]
* 12:57 _joe_: rebooting appservers in eqiad, 3 at a time
* 12:57 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 12:37 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 12:34 kormat: deploying wmfmariadbpy 0.4
* 12:21 jayme@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 11:53 XioNoX: add new icinga hosts to mr policies - [[phab:T260533|T260533]]
* 11:40 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 11:36 Lucas_WMDE: EU backport&config done
* 11:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620888{{!}}Add Wikisource wordmark for trwikisource (T260658)]], part 2 (duration: 00m 55s)
* 11:32 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf '%s\n' 'https://en.wikipedia.org/static/images/mobile/copyright/wikisource-wordmark-tr.svg' {{!}} mwscript purgeList.php # [[phab:T260658|T260658]]
* 11:32 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/mobile/copyright/wikisource-wordmark-tr.svg: Config: [[gerrit:620888{{!}}Add Wikisource wordmark for trwikisource (T260658)]], part 1 (duration: 00m 55s)
* 11:24 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:595543{{!}}Enable Data Bridge on Catalan Wikipedia (T232584)]] (duration: 01m 01s)
* 11:06 jbond42: deploy net-snmp update to buster
* 10:56 oblivian@cumin1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw,name=mw229.*
* 10:55 oblivian@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97)
* 10:54 marostegui: Reboot db2125 after running a full upgrade - [[phab:T260670|T260670]]
* 10:46 marostegui: Powercycle db2125 from the idrac [[phab:T260670|T260670]]
* 10:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2125 - host down [[phab:T260670|T260670]]', diff saved to https://phabricator.wikimedia.org/P12288 and previous config saved to /var/cache/conftool/dbconfig/20200818-100718-marostegui.json
* 09:45 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 09:43 jiji@cumin1001: conftool action : set/pooled=yes; selector: name=mw2250.codfw.wmnet
* 09:40 oblivian@cumin1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw,name=mw214[234].*
* 09:40 oblivian@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97)
* 09:35 kart_: Update cxserver to 2020-08-17-090424-production ([[phab:T259980|T259980]])
* 09:32 kartik@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' .
* 09:29 kartik@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' .
* 09:28 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 09:28 oblivian@cumin1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw,name=mw214[02].*
* 09:26 volans: upgraded spicerack to v0.0.39 on cumin hosts
* 09:25 kartik@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
* 09:21 volans: uploaded spicerack_0.0.39-1+deb10u1 to apt.wikimedia.org buster-wikimedia
* 09:05 hashar: Restarting CI Jenkins
* 08:44 vgutierrez: restart ats-tls on cp5006
* 08:24 oblivian@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97)
* 08:17 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 08:16 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 08:10 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 08:02 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1089', diff saved to https://phabricator.wikimedia.org/P12284 and previous config saved to /var/cache/conftool/dbconfig/20200818-080256-marostegui.json
* 07:58 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 07:53 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 07:45 godog: VictorOps ack'd incidents will re-trigger after 24h if not resolved - [[phab:T259465|T259465]]
* 07:44 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1)
* 07:43 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P12283 and previous config saved to /var/cache/conftool/dbconfig/20200818-074325-marostegui.json
* 07:42 _joe_: performing rolling reboot of all codfw api servers
* 07:38 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P12282 and previous config saved to /var/cache/conftool/dbconfig/20200818-072349-marostegui.json
* 07:19 oblivian@cumin1001: conftool action : set/pooled=yes; selector: name=mw213[5-9].codfw.wmnet
* 07:16 jynus: update rest of phabricator passwords [[phab:T250361|T250361]]
* 07:11 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1089', diff saved to https://phabricator.wikimedia.org/P12281 and previous config saved to /var/cache/conftool/dbconfig/20200818-071121-marostegui.json
* 07:08 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 07:07 godog: prometheus eqiad: add 100G to prometheus/global
* 07:01 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 07:01 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 07:01 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 06:53 twentyafterfour: phabricator maintenance successful
* 06:48 jynus: deploy another password change to phabricator service (potentially disruptive) [[phab:T250361|T250361]]
* 06:41 XioNoX: add cloudflare PNI IPs in eqiad - [[phab:T259036|T259036]]
* 06:21 jynus: deploy password change to phabricator service [[phab:T146055|T146055]]
* 06:06 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 06:01 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 05:52 _joe_: running puppet on mc1020 [[phab:T260622|T260622]]
* 05:02 twentyafterfour: phabricator appears to be fully functional
* 05:01 twentyafterfour: phabricator read-only ended
* 05:00 twentyafterfour: phabricator is now read-only
* 05:00 marostegui: Failover m3 (phabricator) database master from db1128 to db1132 - [[phab:T259589|T259589]]
* 04:32 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1088', diff saved to https://phabricator.wikimedia.org/P12279 and previous config saved to /var/cache/conftool/dbconfig/20200818-043241-marostegui.json
* 01:54 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1376.eqiad.wmnet
* 01:38 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1343.eqiad.wmnet
* 01:37 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1344.eqiad.wmnet
* 01:04 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 00:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 00:48 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 00:45 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 00:39 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 00:39 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 00:38 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1341.eqiad.wmnet
* 00:37 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1340.eqiad.wmnet
* 00:37 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1339.eqiad.wmnet
* 00:31 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 00:24 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 00:15 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 00:13 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 00:07 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 00:07 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1315.eqiad.wmnet
* 00:06 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
 
== 2020-08-17 ==
* 23:59 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:58 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:49 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:49 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1313.eqiad.wmnet
* 23:49 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:49 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:47 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 23:41 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:40 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1312.eqiad.wmnet
* 23:40 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 23:30 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:28 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1297.eqiad.wmnet
* 23:26 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 23:25 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:23 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:11 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:11 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
* 23:11 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:10 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:10 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
* 23:00 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 22:55 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:47 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:45 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:43 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:43 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1286.eqiad.wmnet
* 22:43 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 22:41 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 22:37 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:37 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1285.eqiad.wmnet
* 22:33 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 22:26 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:26 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 22:26 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1284.eqiad.wmnet
* 22:25 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 22:23 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 22:21 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 22:21 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' .
* 22:17 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:15 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:09 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:08 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1282.eqiad.wmnet
* 22:04 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 22:02 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:02 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1281.eqiad.wmnet
* 22:00 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 21:57 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=aawiktionary --site-group wiktionary ([[phab:T259360|T259360]])
* 21:56 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 21:56 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' .
* 21:53 ppchelko@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add api-gateway.request stream config [[phab:T259736|T259736]], one host timed out (duration: 00m 55s)
* 21:49 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:48 ppchelko@deploy1001: sync-file aborted: Add api-gateway.request stream config [[phab:T259736|T259736]] (duration: 05m 01s)
* 21:47 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1278.eqiad.wmnet
* 21:46 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 21:43 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:42 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1279.eqiad.wmnet
* 21:42 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 21:38 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Further mitigations for [[phab:T257687|T257687]] (duration: 00m 57s)
* 21:38 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 21:36 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 21:34 effie: blocking temporarily traffic to mc1020
* 21:23 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:22 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:20 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:19 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1276.eqiad.wmnet
* 21:12 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2240.codfw.wmnet
* 21:08 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:04 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 20:54 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 20:47 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:38 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:33 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:27 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:24 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 20:20 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:19 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 20:02 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 19:53 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 19:39 dzahn@cumin1001: START - Cookbook sre.hosts.reboot-single
* 19:30 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 19:28 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 19:22 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 19:01 ppchelko@deploy1001: Finished deploy [restbase/deploy@7f16bad]: Add thankyouwiki [[phab:T259002|T259002]], take 3 (duration: 02m 57s)
* 18:58 ppchelko@deploy1001: Started deploy [restbase/deploy@7f16bad]: Add thankyouwiki [[phab:T259002|T259002]], take 3
* 18:58 ppchelko@deploy1001: Finished deploy [restbase/deploy@7f16bad]: Add thankyouwiki [[phab:T259002|T259002]], take 2 (duration: 11m 19s)
* 18:46 ppchelko@deploy1001: Started deploy [restbase/deploy@7f16bad]: Add thankyouwiki [[phab:T259002|T259002]], take 2
* 18:46 ppchelko@deploy1001: Finished deploy [restbase/deploy@7f16bad]: Add thankyouwiki [[phab:T259002|T259002]] (duration: 131m 17s)
* 18:46 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 18:43 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 18:39 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 18:32 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 18:08 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|808c17d28c5ebf5ed75f70c224d66129eb2edcd8}}: Change logo for lldwiki to match the requested one ([[phab:T259432|T259432]]) (duration: 00m 56s)
* 18:04 urbanecm@deploy1001: Synchronized static/images/project-logos/: {{Gerrit|67e8f886cd1a9cd2b63ed69761bec6c52889a5b6}}: Add logo files for lldwiki ([[phab:T259432|T259432]]) (duration: 00m 56s)
* 17:17 cdanis@cumin1001: conftool action : set/pooled=yes; selector: name=mw1359.*
* 17:06 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 17:04 oblivian@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=codfw,name=mw2246.codfw.wmnet
* 17:01 oblivian@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97)
* 16:36 jynus: restart backup2001, backup1001 one after the other
* 16:35 ppchelko@deploy1001: Started deploy [restbase/deploy@7f16bad]: Add thankyouwiki [[phab:T259002|T259002]]
* 16:31 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 16:27 urbanecm@deploy1001: Synchronized private/PrivateSettings.php: Update [[phab:T250887|T250887]] mitigations (duration: 00m 56s)
* 16:23 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wgEventLoggingSchemas - remove unneeded override for SearchSatisfaction - [[phab:T259163|T259163]] (duration: 00m 56s)
* 16:22 cdanis@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 16:21 oblivian@cumin1001: conftool action : set/pooled=inactive; selector: cluster=jobrunner,dc=codfw,name=mw2250.codfw.wmnet
* 16:20 oblivian@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,dc=codfw
* 16:20 cdanis@cumin1001: START - Cookbook sre.hosts.downtime
* 16:14 cdanis@cumin1001: conftool action : set/pooled=inactive; selector: name=mw1359.*
* 16:12 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
* 16:12 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 15:44 ppchelko@deploy1001: Finished deploy [restbase/deploy@ddcecce]: [[phab:T257943|T257943]] [[phab:T260556|T260556]] [[phab:T253478|T253478]] [[phab:T254490|T254490]] [[phab:T259054|T259054]]. take 3. feeds timed out (duration: 01m 31s)
* 15:43 ppchelko@deploy1001: Started deploy [restbase/deploy@ddcecce]: [[phab:T257943|T257943]] [[phab:T260556|T260556]] [[phab:T253478|T253478]] [[phab:T254490|T254490]] [[phab:T259054|T259054]]. take 3. feeds timed out
* 15:43 ppchelko@deploy1001: Finished deploy [restbase/deploy@ddcecce]: [[phab:T257943|T257943]] [[phab:T260556|T260556]] [[phab:T253478|T253478]] [[phab:T254490|T254490]] [[phab:T259054|T259054]]. take 2. feeds timed out (duration: 20m 40s)
* 15:36 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕦☕ homer 'cr*'  commit 'revert skipping RPKI validation for Jio AS55836 {{Gerrit|I0fd4683}} [[phab:T260452|T260452]]'
* 15:30 cdanis: ❌cdanis@cumin1001.eqiad.wmnet ~ 🕦☕ homer 'cr*-codfw*'  commit 'revert skipping RPKI validation for Jio AS55836 {{Gerrit|I0fd4683}} [[phab:T260452|T260452]]'
* 15:22 ppchelko@deploy1001: Started deploy [restbase/deploy@ddcecce]: [[phab:T257943|T257943]] [[phab:T260556|T260556]] [[phab:T253478|T253478]] [[phab:T254490|T254490]] [[phab:T259054|T259054]]. take 2. feeds timed out
* 15:22 ppchelko@deploy1001: Finished deploy [restbase/deploy@ddcecce]: [[phab:T257943|T257943]] [[phab:T260556|T260556]] [[phab:T253478|T253478]] [[phab:T254490|T254490]] [[phab:T259054|T259054]] (duration: 02m 30s)
* 15:19 ppchelko@deploy1001: Started deploy [restbase/deploy@ddcecce]: [[phab:T257943|T257943]] [[phab:T260556|T260556]] [[phab:T253478|T253478]] [[phab:T254490|T254490]] [[phab:T259054|T259054]]
* 15:08 ppchelko@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'changeprop' for release 'production' .
* 15:06 ppchelko@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'changeprop' for release 'production' .
* 15:04 ppchelko@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' .
* 14:57 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wgEventLoggingSchemas - schema revision version bump for erroring schemas - all wikis (take 2) - [[phab:T254606|T254606]] (duration: 00m 53s)
* 14:57 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
* 14:57 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-cluster
* 14:44 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wgEventLoggingSchemas - schema revision version bump for erroring schemas - all wikis - [[phab:T254606|T254606]] (duration: 00m 55s)
* 14:35 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wgEventLoggingSchemas - schema revision version bump for erroring schemas - group0 - [[phab:T254606|T254606]] (duration: 00m 56s)
* 14:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1088 after upgrading its mysql package', diff saved to https://phabricator.wikimedia.org/P12277 and previous config saved to /var/cache/conftool/dbconfig/20200817-141449-marostegui.json
* 14:09 marostegui: Sanitize thankyouwiki on db1124:3315, db2094:3315 - [[phab:T260551|T260551]]
* 14:03 marostegui: Sanitize lldwiki on db1124:3315 and db2094:3315 [[phab:T259436|T259436]]
* 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1088 after upgrading its mysql package', diff saved to https://phabricator.wikimedia.org/P12276 and previous config saved to /var/cache/conftool/dbconfig/20200817-140229-marostegui.json
* 13:58 Amir1: start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https ([[phab:T259432|T259432]])
* 13:54 Urbanecm: Creating thankyouwiki and lldwiki is done
* 13:54 urbanecm@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 01m 52s)
* 13:54 Urbanecm: Create account Pcoombe (WMF) at thankyouwiki, email set to pcoombe@wikimedia.org ([[phab:T259002|T259002]])
* 13:52 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating thankyouwiki ([[phab:T259002|T259002]]) (duration: 00m 55s)
* 13:51 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating thankyouwiki ([[phab:T259002|T259002]]) (duration: 00m 55s)
* 13:49 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating thankyouwiki ([[phab:T259002|T259002]])
* 13:48 urbanecm@deploy1001: Synchronized dblists: Creating thankyouwiki ([[phab:T259002|T259002]]) (duration: 00m 55s)
* 13:47 marostegui: Deploy MCR change on db1104
* 13:47 urbanecm@deploy1001: Synchronized wmf-config/db-codfw.php: Creating thankyouwiki ([[phab:T259002|T259002]]) (duration: 00m 56s)
* 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1104 for MCR change', diff saved to https://phabricator.wikimedia.org/P12275 and previous config saved to /var/cache/conftool/dbconfig/20200817-134701-marostegui.json
* 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P12274 and previous config saved to /var/cache/conftool/dbconfig/20200817-134619-marostegui.json
* 13:46 urbanecm@deploy1001: Synchronized wmf-config/db-eqiad.php: Creating thankyouwiki ([[phab:T259002|T259002]]) (duration: 00m 55s)
* 13:46 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1088 after upgrading its mysql package', diff saved to https://phabricator.wikimedia.org/P12273 and previous config saved to /var/cache/conftool/dbconfig/20200817-134604-marostegui.json
* 13:41 jayme: imported td-agent-bit_1.5.3-0 to buster-wikimedia - [[phab:T260536|T260536]]
* 13:40 jayme: imported !log imported to buster-wikimedia
* 13:39 marostegui: Upgrade db1088 (s6) to a newer mysql version (10.4.14)
* 13:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1088 for mysql upgrade', diff saved to https://phabricator.wikimedia.org/P12272 and previous config saved to /var/cache/conftool/dbconfig/20200817-133905-marostegui.json
* 13:34 jbond42: deploy json-c security update to buster
* 13:33 marostegui: Restart mysql on db2102 (testing new package)
* 13:30 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P12271 and previous config saved to /var/cache/conftool/dbconfig/20200817-133043-marostegui.json
* 13:29 urbanecm@deploy1001: Synchronized langlist: Creating lldwiki ([[phab:T259432|T259432]]) (duration: 00m 54s)
* 13:28 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Creating lldwiki ([[phab:T259432|T259432]]) (duration: 00m 55s)
* 13:27 urbanecm@deploy1001: sync-file aborted: Creating lldwiki ([[phab:T259432|T259432]])¨ (duration: 00m 00s)
* 13:26 urbanecm@deploy1001: Synchronized static/images/project-logos/: Creating lldwiki ([[phab:T259432|T259432]]) (duration: 00m 53s)
* 13:25 urbanecm@deploy1001: rebuilt and synchronized wikiversions files: Creating lldwiki ([[phab:T259432|T259432]])
* 13:23 urbanecm@deploy1001: Synchronized dblists: Creating lldwiki ([[phab:T259432|T259432]]) (duration: 00m 56s)
* 13:22 urbanecm@deploy1001: Synchronized wmf-config/db-codfw.php: Creating lldwiki ([[phab:T259432|T259432]]) (duration: 00m 56s)
* 13:20 urbanecm@deploy1001: Synchronized wmf-config/db-eqiad.php: Creating lldwiki ([[phab:T259432|T259432]]) (duration: 00m 55s)
* 13:13 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P12270 and previous config saved to /var/cache/conftool/dbconfig/20200817-131307-marostegui.json
* 13:10 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 13:10 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 13:09 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P12269 and previous config saved to /var/cache/conftool/dbconfig/20200817-130127-marostegui.json
* 12:58 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:53 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:53 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:53 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:53 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depoool db1089 for MCR change', diff saved to https://phabricator.wikimedia.org/P12268 and previous config saved to /var/cache/conftool/dbconfig/20200817-124458-marostegui.json
* 12:44 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P12267 and previous config saved to /var/cache/conftool/dbconfig/20200817-124409-marostegui.json
* 12:44 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:38 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:35 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 12:35 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 12:35 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:27 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:22 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P12266 and previous config saved to /var/cache/conftool/dbconfig/20200817-122234-marostegui.json
* 12:21 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:20 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:19 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:19 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P12265 and previous config saved to /var/cache/conftool/dbconfig/20200817-121600-marostegui.json
* 12:05 Lucas_WMDE: EU backport window done
* 12:02 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php bjnwiki --fix {{!}} tee [[phab:T259429|T259429]]-fix
* 12:02 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php bjnwiki {{!}} tee [[phab:T259429|T259429]]-dryrun
* 12:01 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620643{{!}}Set Portal and Portal_talk namespaces in bjnwiki as an extra namespace. (T259429)]] (duration: 00m 55s)
* 11:57 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P12264 and previous config saved to /var/cache/conftool/dbconfig/20200817-115741-marostegui.json
* 11:54 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620513{{!}}Add Wiktionary wordmark for eswiktionary (T254059)]], part 2 (duration: 00m 57s)
* 11:53 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf 'https://en.wikipedia.org/static/images/mobile/copyright/wiktionary-wordmark-es.svg\n' {{!}} mwscript purgeList.php # [[phab:T254059|T254059]]
* 11:53 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/mobile/copyright/wiktionary-wordmark-es.svg: Config: [[gerrit:620513{{!}}Add Wiktionary wordmark for eswiktionary (T254059)]], part 1 (duration: 00m 56s)
* 11:46 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf 'https://en.wikipedia.org/static/images/project-logos/zh_classicalwiki%s.png\n' '' '-1.5x' '-2x' {{!}} mwscript purgeList.php # [[phab:T259006|T259006]]
* 11:45 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: Config: [[gerrit:620510{{!}}Change the logo of lzh Wikipedia (T259006)]] (duration: 00m 55s)
* 11:40 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620509{{!}}Add Turkish powered by MW and Wikimedia project icons for Turkish Wikiquote, Turkish Wiktionary, Turkish Wikisource and Turkish Wikibooks (T260493)]] (duration: 00m 55s)
* 11:35 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620507{{!}}Add Turkish powered by MW and Wikimedia project icons (T260492)]] (duration: 00m 57s)
* 11:25 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:14 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 11:12 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:10 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 11:09 cparle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [SDC] configure mediasearch A/B test (duration: 01m 08s)
* 11:08 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 10:57 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:54 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:52 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:52 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:52 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:51 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 10:49 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:47 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:42 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:38 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:36 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:35 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:35 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:30 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 10:29 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 10:29 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 10:14 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:14 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 10:13 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:13 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:10 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:07 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:06 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 09:58 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:58 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:56 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:55 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 09:52 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:48 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 09:45 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 09:42 jynus: updating compiler facts for cloud puppet compiler project to include new host dbprov2003
* 09:39 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 09:38 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:38 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:36 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 09:36 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 09:29 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:28 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:27 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:23 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 09:22 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:21 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 09:18 _joe_: running a full apt-get upgrade on mw1379-1380
* 09:18 _joe_: re-upgrading imagemagick on mw1378
* 09:16 _joe_: upgrading packages on mw1377
* 09:14 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 09:06 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:06 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:05 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 08:25 jayme: forcing a puppet run on all mw-api servers in eqiad - [[phab:T260329|T260329]]
* 07:52 _joe_: repooling mw1382
* 07:37 _joe_: running the same test on mw1382 [[phab:T260329|T260329]]
* 07:34 _joe_: repooling mw1381
* 07:15 _joe_: running the same test on mw1381 [[phab:T260329|T260329]]
* 07:15 _joe_: repooled mw1281
* 06:26 _joe_: stop testing on mw1281, [[phab:T260329|T260329]]
* 05:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 05:43 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 05:28 marostegui: Stop mysql on db1099:3311, db1099:3318 for reimage
* 05:28 _joe_: depooling mw1281 for testing for [[phab:T260329|T260329]]
* 05:25 marostegui: Deploy schema change on db1139:3311
* 05:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1099:3311, db1099:3318 for reimage and MCR change', diff saved to https://phabricator.wikimedia.org/P12263 and previous config saved to /var/cache/conftool/dbconfig/20200817-052147-marostegui.json
 
== 2020-08-16 ==
* 11:12 gehel: repooling wdqs1004 - catched up on lag
 
== 2020-08-15 ==
* 21:18 gehel: depooling wdqs1004 and restarting services, will wait to catch up on lag before repooling
 
== 2020-08-14 ==
* 19:41 effie: restart mwdebug1002
* 16:58 cdanis: done deploying 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8' to all routers [[phab:T260449|T260449]]
* 16:44 cdanis: ❌cdanis@cumin1001.eqiad.wmnet ~ 🕧☕ homer 'cr2-esams*'  commit 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8'
* 16:39 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕧☕ homer 'cr1-codfw*'  commit 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8'
* 16:36 cdanis: ❌cdanis@cumin1001.eqiad.wmnet ~ 🕧☕ homer 'cr2-codfw*'  commit 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8'
* 02:41 eileen: tools revision changed from {{Gerrit|9a89f45974}} to {{Gerrit|b4ebd1e564}}
 
== 2020-08-13 ==
* 23:39 tzatziki: removing 3 files for legal compliance
* 22:03 mutante: switching xhgui from tungsten to xhgui1001 - ran puppet on webperf*001 - [[phab:T180761|T180761]] [[phab:T158837|T158837]]
* 21:54 andrew@deploy1001: Finished deploy [horizon/deploy@f3dcb29]: fix proxy in project-local domain --bug [[phab:T260388|T260388]] (duration: 03m 53s)
* 21:50 andrew@deploy1001: Started deploy [horizon/deploy@f3dcb29]: fix proxy in project-local domain --bug [[phab:T260388|T260388]]
* 21:11 mutante: rsyncing /var/lib/jenkins from releases1001 to releases1002 and then all other releases* servers. 57GB, overwriting existing data from manual config ([[phab:T247652|T247652]])
* 20:53 kormat: dropping xhgui.xhgui on m2
* 19:35 thcipriani@deploy1001: Synchronized php-1.36.0-wmf.4/extensions/DiscussionTools: [[gerrit:620030{{!}}Revert new reply API (again)]] [[phab:T259855|T259855]] (duration: 00m 57s)
* 18:06 herron: restarted ES on logstash1010
* 18:05 dpifke@deploy1001: Synchronized wmf-config/ProductionServices.php: Enabling new XHGui backend ([[phab:T180761|T180761]]) (duration: 00m 56s)
* 17:16 hnowlan: deployed ATS and varnish rules to route api.wikimedia.org
* 16:26 hnowlan: created api.wikimedia.org
* 15:49 hnowlan: moving api-gateway service to state production. critical set to false
* 15:41 herron: restart ES on logstash1012
* 14:56 fdans@deploy1001: Finished deploy [analytics/refinery@ba1a439]: Regular analytics weekly train (duration: 11m 34s)
* 14:45 ema: repool mw1382 with kernel memory accounting disabled [[phab:T260281|T260281]]
* 14:45 fdans@deploy1001: Started deploy [analytics/refinery@ba1a439]: Regular analytics weekly train
* 14:41 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 14:40 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 14:38 ema: reboot mw1382 with kernel memory accounting disabled [[phab:T260281|T260281]]
* 14:34 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:34 _joe_: rebooting mw1381 with a newer kernel, mw1383 as control with the old kernel [[phab:T260329|T260329]]
* 14:33 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:31 _joe_: installing kernel 4.19.0-0.bpo.9 on mw1381 [[phab:T260329|T260329]]
* 14:05 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 14:00 elukey: create schema[12]00[34] in ganeti - [[phab:T260347|T260347]]
* 13:59 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
* 13:58 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 13:53 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
* 13:51 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 13:46 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
* 13:45 hnowlan: moving api-gateway service to monitoring_setup
* 13:44 elukey@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
* 13:44 hashar: Gracefully restarting Zuul
* 13:39 elukey@cumin1001: START - Cookbook sre.ganeti.makevm
* 13:10 _joe_: forcing a puppet run on the api appservers in eqiad  [[phab:T260329|T260329]]
* 13:07 oblivian@deploy1001: Synchronized wmf-config/CommonSettings.php: revert enabling of lilypond (again) [[phab:T257091|T257091]] [[phab:T260329|T260329]] (duration: 00m 59s)
* 11:24 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:20 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 11:09 hnowlan: restarting pybal on lvs2010 [[phab:T254908|T254908]]
* 11:09 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:06 hnowlan: restarting pybal on lvs2009 [[phab:T254908|T254908]]
* 11:05 hnowlan: restarting pybal on lvs1016 [[phab:T254908|T254908]]
* 11:05 jayme: depool mw1380 for downgrade of poppler-utils,libpoppler-glib8,libpoppler64,curl,libcurl3,libcurl3-gnutls,libpython3.5,python3.5,libpython3.5-stdlib,python3.5-minimal,libpython3.5-minimal,imagemagick-6-common,libmagickcore-6.q16-3,libmagickwand-6.q16-3,imagemagick-6.q16,imagemagick,e2fslibs,e2fsprogs,libcomerr2,libss2 and reboot - [[phab:T260329|T260329]]
* 11:05 hnowlan: restarting pybal on lvs1015 [[phab:T254908|T254908]]
* 11:04 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:42 hnowlan: Moving api-gateway service to from service_setup to lvs_setup and running puppet on LVS servers
* 10:17 jayme: depool mw1379 for downgrade of poppler-utils,libpoppler-glib8,libpoppler64,curl,libcurl3,libcurl3-gnutls,libpython3.5,python3.5,libpython3.5-stdlib,python3.5-minimal,libpython3.5-minimal,imagemagick-6-common,libmagickcore-6.q16-3,libmagickwand-6.q16-3,imagemagick-6.q16,imagemagick,e2fslibs,e2fsprogs,libcomerr2,libss2 and reboot - [[phab:T260329|T260329]]
* 10:04 XioNoX: re-order OSPF interfaces on all routers (now partially netbox driven)
* 09:37 ayounsi@deploy1001: Finished deploy [homer/deploy@89636df]: Homer release v0.2.5 (duration: 03m 03s)
* 09:34 ayounsi@deploy1001: Started deploy [homer/deploy@89636df]: Homer release v0.2.5
* 08:59 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 08:58 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 08:55 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1082', diff saved to https://phabricator.wikimedia.org/P12247 and previous config saved to /var/cache/conftool/dbconfig/20200813-085547-marostegui.json
* 08:45 _joe_: downgrading imagemagick on mw1378 [[phab:T260329|T260329]]
* 08:43 _joe_: downgrading imagemagick on mw1378 [[phab:T260281|T260281]]
* 08:38 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 08:38 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 07:55 _joe_: downgrading curl/libcurl3/libcurl3-gnutls on mw1377 [[phab:T260329|T260329]]
* 07:45 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12246 and previous config saved to /var/cache/conftool/dbconfig/20200813-074528-marostegui.json
* 07:19 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12244 and previous config saved to /var/cache/conftool/dbconfig/20200813-071943-marostegui.json
* 07:16 marostegui: Stop replication on db1082 to remove triggers on sanitarium for MCR changs
* 07:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P12243 and previous config saved to /var/cache/conftool/dbconfig/20200813-071545-marostegui.json
* 06:48 marostegui: Deploy MCR change on dbstore1003:3311
* 06:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 06:01 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1126', diff saved to https://phabricator.wikimedia.org/P12242 and previous config saved to /var/cache/conftool/dbconfig/20200813-060135-marostegui.json
* 06:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 05:43 marostegui: Stop MySQL on db2135 (codfw master), haproxy irc alert will fire [[phab:T260324|T260324]]
* 05:28 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12241 and previous config saved to /var/cache/conftool/dbconfig/20200813-052859-marostegui.json
* 05:12 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12240 and previous config saved to /var/cache/conftool/dbconfig/20200813-051222-marostegui.json
* 05:01 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12239 and previous config saved to /var/cache/conftool/dbconfig/20200813-050107-marostegui.json
* 02:56 mutante: testreduce1001 - systemctl reset-failed ; fix parsoid-vd systemd state and icinga alert
* 00:37 mutante: removing jenkins_service_running checks from secondary servers where it's stopped, manually from icinga config, running puppet on icinga
* 00:14 mutante: re-enabling puppet on releases* servers
 
== 2020-08-12 ==
* 23:44 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:41 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:40 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:39 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:37 wkandek: reboot mw1372
* 23:36 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:36 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:32 wkandek: reboot mw1373
* 23:32 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:31 wkandek: reboot mw1371
* 23:31 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:31 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:30 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:28 wkandek: reboot mw1384
* 23:27 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:27 wkandek: reboot mw1385
* 23:26 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:25 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:24 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:22 wkandek: reboot mw1370
* 23:22 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:19 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:18 wkandek: reboot mw1369
* 23:18 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:17 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:17 wkandek: reboot mw1387
* 23:16 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:16 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:16 wkandek: reboot mw1389
* 23:15 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:14 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:09 wkandek: reboot mw1368
* 23:09 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:08 wkandek: reboot me1367
* 23:08 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:07 wkandek: reboot mw1391
* 23:07 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:06 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:06 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:06 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:05 ejegg: updated Fundraising CiviCRM from {{Gerrit|72452e28a9}} to {{Gerrit|f5469d0a4c}}
* 23:05 wkandek: reboot mw1393
* 23:04 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:04 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 23:01 wkandek: reboot mw1395
* 23:01 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 23:00 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:53 wkandek: reboot mw1397
* 22:53 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:52 wkandek: reboot mw1366
* 22:52 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:52 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:52 wkandek: reboot me1365
* 22:51 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:51 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:51 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:47 wkandek: reboot mw1399
* 22:47 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:46 wkandek: reboot mw1364
* 22:46 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:45 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:44 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:42 wkandek: reboot mw1401
* 22:42 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:41 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:41 wkandek: reboot mw1355
* 22:40 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:40 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:38 wkandek: reboot mw1354
* 22:38 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:36 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:36 wkandek: reboot mw1396
* 22:36 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:35 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:32 wkandek: reboot mw1353
* 22:32 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:31 wkandek: reboot mw1352
* 22:31 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:31 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:30 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:29 wkandek: reboot mw1348
* 22:29 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:28 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:26 wkandek: reboot 1347
* 22:26 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:25 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:23 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:22 wkandek: reboot mw1350
* 22:22 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:21 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:20 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:19 wkandek: reboot mw1346
* 22:19 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:18 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:14 wkandek: reboot mw1345
* 22:13 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:12 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:12 wkandek: reboot mw1349
* 22:12 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:11 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:08 wkandek: reboot mw1333
* 22:07 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:07 wkandek@cumin1001: conftool action : set/pooled=yes; selector: name=mw1330.eqiad.wmnet
* 22:03 wkandek: reboot mw1344
* 22:03 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:02 wkandek: reboot mw1343
* 22:02 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 22:02 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 22:00 wkandek: reboot mw1332
* 22:00 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:56 wkandek@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 21:55 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:53 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:50 wkandek: reboot mw1331
* 21:50 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:48 wkandek: reboot mw1342
* 21:47 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:46 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:46 wkandek@cumin1001: conftool action : set/pooled=yes; selector: name=mw1340.eqiad.wmnet
* 21:41 wkandek@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 21:40 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:39 wkandek@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:39 wkandek: reboot mw1341
* 21:39 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:37 wkandek@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97)
* 21:37 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:36 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:33 wkandek: reboot mw1329
* 21:33 wkandek: reboot mw1328
* 21:32 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:32 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:29 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:28 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:28 ejegg: updated payments-wiki from {{Gerrit|77ff5d70fc}} to {{Gerrit|a7ee1790e0}}
* 21:25 wkandek: reboot mw1340
* 21:25 wkandek@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:23 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:21 wkandek: reboot mw1339
* 21:20 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:20 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:15 wkandek: reboot mw1327
* 21:15 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:14 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:13 wkandek: reboot mw1326
* 21:13 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:11 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:11 wkandek: reboot mw1317
* 21:11 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:10 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:05 wkandek: reboot mw1316
* 21:04 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:03 wkandek: reboot mw1325
* 21:03 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:02 wkandek: reboot mw1324
* 21:02 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:02 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:01 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:01 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 21:01 wkandek: reboot mw1315
* 21:01 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 21:00 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:57 wkandek: reboot mw1323
* 20:57 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:54 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:52 wkandek: reboot mw1322
* 20:52 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:51 wkandek: reboot mw1314
* 20:51 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:50 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:50 wkandek: reboot mw1313
* 20:50 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:49 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:48 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:44 wkandek: reboot mw1312
* 20:44 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:43 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:43 wkandek: reboot mw1321
* 20:42 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:41 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:40 wkandek: reboot mw1297
* 20:40 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:39 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:39 wkandek: reboot mw1320
* 20:39 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:38 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:34 wkandek: reboot mw1290
* 20:34 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:33 wkandek: reboot mw1319
* 20:33 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:32 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:31 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:29 wkandek: reboot mw1275
* 20:29 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:28 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:26 wkandek: reboot mw1289
* 20:25 wkandek: reboot mw1288
* 20:25 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:25 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:24 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:23 wkandek: reboot mw1274
* 20:23 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:23 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:22 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:20 wkandek: reboot mw1273
* 20:20 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:16 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 20:13 wkandek: reboot mw1287
* 20:13 wkandek: reboot mw1286
* 20:13 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:13 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:11 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:11 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 20:11 wkandek: reboot mw1272
* 20:11 wkandek: reboot mw1271
* 19:41 hashar: Upgrading Jenkins on contint2001 (primary)
* 19:25 hashar: contint1001: sudo systemctl mask jenkins  # spare server
* 19:25 mutante: all releases* servers except 1001 - disable puppet; stop jenkins, mask jenkins
* 19:22 mutante: releases1002 - stopped and masked jenkins service
* 19:22 mutante: releases2001 - stopped and masked jenkins service
* 19:20 mutante: upgrading jenkins on releases*001
* 19:19 hashar: Upgrading Jenkins on contint1001 (spare)
* 19:16 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.4
* 19:13 mutante: uploade new jenkins version to APT repo; upgrading jenkins on releases1002/2002
* 19:08 effie: pool mw1396
* 19:06 effie: repool mw1395 mw1397 mw1399
* 18:56 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 18:55 root@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 18:50 ladsgroup@deploy1001: Synchronized php-1.36.0-wmf.4/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php: [[phab:T255305{{!}}Set caching of CachingEntityRevisionLookup to CACHE_NONE in client]] (duration: 02m 13s)
* 18:47 wkandek: reboot mw1270
* 18:47 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:45 wkandek: reboot mw1269
* 18:41 root@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:39 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 18:39 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 18:38 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 18:28 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 18:25 wkandek: reboot mw1268
* 18:25 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 18:25 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 18:23 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:22 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 18:22 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 18:22 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:22 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:22 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:17 jiji@cumin1001: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97)
* 18:17 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 18:16 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 18:10 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable GrowthExperiments on hewiki ([[phab:T255020|T255020]]) (duration: 01m 03s)
* 18:08 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 18:07 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 18:07 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 18:06 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 18:04 ladsgroup@deploy1001: Synchronized php-1.36.0-wmf.4/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: [[phab:T255305{{!}}Set caching of CachingEntityRevisionLookup to CACHE_NONE in repo]] (duration: 01m 06s)
* 18:02 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 18:02 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 18:00 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:59 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 17:58 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 17:56 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 17:52 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 17:52 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 17:52 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:51 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 17:51 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 17:51 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:50 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:49 effie: reboot mw1265 mw1282 mw1283
* 17:45 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:45 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 17:37 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 17:36 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 17:30 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 17:25 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 17:21 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:21 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:20 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:20 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 17:19 effie: reboot mw1263 mw1264 mw1279 and mw1281
* 17:17 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 17:17 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 17:16 cdanis: for posterity: mw1359 has a bunch of special packages installed (previously recorded in SAL) and also has `sudo memleak-bpfcc -o 60000 -z 31 -Z 33 30` running in a tmux in an attempt to understand what's causing the page fragmentation in the appserver fleet
* 17:16 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 17:16 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 17:15 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 17:15 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 17:13 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' .
* 17:13 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 17:03 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 17:00 jiji@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 16:57 sbassett@deploy1001: Synchronized private/PrivateSettings.php: Additional mitigations for [[phab:T257687|T257687]] (duration: 01m 03s)
* 16:53 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:52 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:52 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 16:50 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:49 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 16:48 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 16:45 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 16:45 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 16:44 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 16:35 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:35 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 16:32 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 16:31 effie: reboot mw1277 mw1278 && mw1261 mw1262
* 16:29 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
* 16:24 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 16:04 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 16:04 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|I3726a6364d}}, [[phab:T257079|T257079]] (duration: 01m 02s)
* 15:56 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 15:52 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:50 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:48 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:48 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:42 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:37 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:36 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:32 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:32 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:32 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:31 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 15:26 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:25 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:22 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 15:21 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:19 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:19 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:16 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:16 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:15 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:14 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:12 cdanis: ✔️ cdanis@mw1359.eqiad.wmnet ~ 🕚☕ sudo apt install linux-headers-4.9.0-12-amd64
* 15:10 cdanis: ✔️ cdanis@mw1359.eqiad.wmnet ~ 🕚☕ sudo apt install python3-netaddr ieee-data
* 15:09 cdanis: ✔️ cdanis@mw1359.eqiad.wmnet ~ 🕚☕ sudo dpkg -i bpfcc-tools_0.12.0-2_all.deb libbpfcc_0.12.0-2_amd64.deb python3-bpfcc_0.12.0-2_all.deb
* 15:08 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 15:03 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 15:03 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 15:03 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 15:03 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:59 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:54 cdanis: again un-kludging deneb.codfw.wmnet:/var/cache/pbuilder/hooks/stretch/D02backports
* 14:53 jayme@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 14:52 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:45 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 14:44 cdanis: temporarily re-kludging deneb.codfw.wmnet:/var/cache/pbuilder/hooks/stretch/D02backports, original in my homedir
* 14:37 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:37 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 14:35 cdanis: un-kludging deneb.codfw.wmnet:/var/cache/pbuilder/hooks/stretch/D02backports
* 14:32 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 14:31 cdanis: temporarily kludging deneb.codfw.wmnet:/var/cache/pbuilder/hooks/stretch/D02backports, original in my homedir
* 14:24 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 14:24 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 14:02 kormat: uploaded wmfmariadbpy 0.3 to apt
* 13:57 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 13:56 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 13:48 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 13:48 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 13:43 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 13:43 jiji@cumin1001: START - Cookbook sre.hosts.reboot-single
* 13:42 effie: restart mw1383 & mw1386
* 13:41 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 13:41 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 13:27 hashar@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.4 (duration: 01m 16s)
* 13:25 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.4
* 13:20 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 13:19 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 13:15 cdanis: ✔️ cdanis@mw1357.eqiad.wmnet ~ 🕘☕ sudo sysctl -w vm/compact_memory=1
* 13:12 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 13:07 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 13:04 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:59 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:52 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:50 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:33 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1)
* 12:27 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:20 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:20 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 12:16 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:15 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 12:15 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
* 12:15 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:51 ema: pool mw1363 after reboot
* 11:49 jynus: creating artificial low replication lag on db2130 to test icinga alerts [[phab:T253120|T253120]]
* 11:41 ema@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:37 ema@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:30 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:28 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:25 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:24 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:21 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:17 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:17 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:13 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 11:10 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:08 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:07 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
* 11:07 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 11:00 oblivian@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99)
* 11:00 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:55 _joe_: rebooting mw1361
* 10:51 jayme: rebooting mw1356
* 10:49 _joe_: rebooting mw1378
* 09:45 _joe_: repooling mw1377
* 09:40 _joe_: rebooting mw1377
* 09:22 _joe_: depool mw1357 tool
* 09:14 _joe_: depooling mw1377 for inspection
* 09:12 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1110', diff saved to https://phabricator.wikimedia.org/P12220 and previous config saved to /var/cache/conftool/dbconfig/20200812-091211-marostegui.json
* 09:08 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1110', diff saved to https://phabricator.wikimedia.org/P12219 and previous config saved to /var/cache/conftool/dbconfig/20200812-090831-marostegui.json
* 08:50 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1110', diff saved to https://phabricator.wikimedia.org/P12218 and previous config saved to /var/cache/conftool/dbconfig/20200812-085021-marostegui.json
* 08:35 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1110', diff saved to https://phabricator.wikimedia.org/P12217 and previous config saved to /var/cache/conftool/dbconfig/20200812-083548-marostegui.json
* 07:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 07:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 07:31 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1110 for reimage', diff saved to https://phabricator.wikimedia.org/P12215 and previous config saved to /var/cache/conftool/dbconfig/20200812-073130-marostegui.json
* 04:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1126 for MCR change', diff saved to https://phabricator.wikimedia.org/P12214 and previous config saved to /var/cache/conftool/dbconfig/20200812-045157-marostegui.json
 
== 2020-08-11 ==
* 23:41 Urbanecm: Evening B&C window completed
* 23:39 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0f238f71c95c7bd7534c28abfac759fbb47f674f}}: Update wgMFRemovableClasses ([[phab:T231160|T231160]]) (duration: 01m 03s)
* 23:36 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/MobileFrontend/extension.json: {{Gerrit|c22d65ff9b2439f484ab8ccffed87b00e78c3ad2}}: Hide vertical nav-boxes on mobile domain ([[phab:T231160|T231160]]) (duration: 01m 03s)
* 23:34 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.4/extensions/MobileFrontend/extension.json: {{Gerrit|81d54b0ec82d0b78f723f9400031e918a4a143aa}}: Hide vertical nav-boxes on mobile domain ([[phab:T231160|T231160]]) (duration: 01m 05s)
* 23:07 urbanecm@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|28faa279dacf6a4d6f0a663844e913738c2fa142}}: Switching to updated license definition (duration: 01m 04s)
* 21:52 krinkle@deploy1001: Synchronized php-1.36.0-wmf.3/includes/skins/SkinMustache.php: {{Gerrit|Ibe1f07346}}, [[phab:T259872|T259872]], [[phab:T259858|T259858]] (duration: 01m 04s)
* 19:40 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 19:40 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 19:35 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add streams for eventgate-main - [[phab:T251935|T251935]] (duration: 01m 04s)
* 19:21 ejegg: updated payments-wiki from {{Gerrit|f199c071c3}} to {{Gerrit|77ff5d70fc}}
* 18:55 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 18:48 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 18:44 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' .
* 18:44 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 18:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Grant investigate right to checkuser group on frwiki ([[phab:T260171|T260171]]) (duration: 01m 04s)
* 18:18 ppchelko@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: Beta-only: Configured additional settings for API Portal beta wiki gerrit:619339 (duration: 01m 03s)
* 18:05 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Direct GrowthExperiments help panel questions to mentors on cswiki ([[phab:T250235|T250235]]) (duration: 01m 03s)
* 17:56 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Remove extraneous mediawiki.api-request stream - [[phab:T251935|T251935]] (duration: 01m 01s)
* 17:53 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
* 17:53 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
* 17:43 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
* 17:43 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
* 17:38 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
* 17:36 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:33 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' .
* 17:31 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:28 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' .
* 17:25 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' .
* 17:04 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:58 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 16:53 hashar@deploy1001: Synchronized php-1.36.0-wmf.4/skins/MinervaNeue/: Revert "ServiceWiring: Avoid usage of deprecated Title::getSubjectPage()" - [[phab:T260155|T260155]] (duration: 01m 06s)
* 16:12 herron: migrating lists.wikimedia.org services from fermium to lists1001 [[phab:T224586|T224586]]
* 15:36 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.4
* 15:27 hashar@deploy1001: Finished scap: (no justification provided) (duration: 30m 51s)
* 14:59 marostegui: Deploy MCR change on db1116:3318
* 14:56 hashar@deploy1001: Started scap: (no justification provided)
* 14:56 hashar@deploy1001: Pruned MediaWiki: 1.36.0-wmf.2 (duration: 04m 15s)
* 14:55 jayme: updated helmfile to 0.125.2-1 on contint* and deploy*
* 14:52 otto@deploy1001: Finished deploy [analytics/refinery@35c4430]: Deploying to an-launcher1002 to get camus wrapper script changes - [[phab:T251935|T251935]] (duration: 01m 14s)
* 14:51 otto@deploy1001: Started deploy [analytics/refinery@35c4430]: Deploying to an-launcher1002 to get camus wrapper script changes - [[phab:T251935|T251935]]
* 14:50 hashar@deploy1001: Pruned MediaWiki: 1.36.0-wmf.1 (duration: 02m 07s)
* 14:48 jayme: imported helmfile_0.125.2-1 to buster-wikimedia, jessie-wikimedia, stretch-wikimedia
* 14:47 hashar@deploy1001: Pruned MediaWiki: 1.35.0-wmf.41 (duration: 04m 20s)
* 14:40 hashar@deploy1001: Pruned MediaWiki: 1.35.0-wmf.40 (duration: 10m 24s)
* 14:37 papaul: replacing msw-b5,b6,b7 and b8
* 14:30 hashar: Cleaning old MediaWiki versions that were never removed
* 14:27 hashar@deploy1001: sync aborted: testwikis wikis to 1.36.0-wmf.4 (duration: 72m 36s)
* 14:10 hashar: mw1319: scap pull
* 13:27 jayme@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
* 13:27 jayme@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 13:23 oblivian@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 13:16 oblivian@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 13:14 hashar@deploy1001: Started scap: testwikis wikis to 1.36.0-wmf.4
* 13:12 hashar: Applied 1.36.0-wmf.4 security patches # [[phab:T257972|T257972]]
* 13:03 jayme@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
* 13:03 jayme@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 12:52 kormat: uploaded wmfmariadbpy 0.2 packages to apt1001
* 12:36 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 12:36 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
* 11:54 marostegui: Install new MariaDB 10.4.14 on db2102
* 11:42 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 11:30 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 11:18 Urbanecm: EU B&C window done
* 11:08 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit{{!}}619255{{!}}Enable ContentTranslation in Sundanese WP as a default tool (T258502)]] (duration: 00m 59s)
* 10:39 volans: migrating *all* eqiad mgmt DNS records to the autogenerated ones via Netbox - [[phab:T233183|T233183]]
* 10:38 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:34 volans@cumin1001: START - Cookbook sre.dns.netbox
* 10:20 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh (exit_code=0)
* 10:01 elukey@cumin1001: START - Cookbook sre.hadoop.change-distro-from-cdh
* 10:00 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0)
* 09:51 elukey@cumin1001: START - Cookbook sre.hadoop.stop-cluster
* 09:29 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:25 volans@cumin1001: START - Cookbook sre.dns.netbox
* 09:11 marostegui: Rename tables on muswiki and mhwiktionary on s3 master (db1123) without replication [[phab:T260112|T260112]]
* 09:01 volans: renewed puppet certificate on scb1001.eqiad.wmnet
* 08:52 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|e6ec237b6b6fb67a0a80613909589bc724f5eecf}}: Revert "Turn muswiki and mhwiktionary to read-only" ([[phab:T259004|T259004]]) (duration: 00m 58s)
* 08:45 urbanecm@deploy1001: Synchronized dblists/: {{Gerrit|81f4594b6c583f938821549b3a1800fec5b120bb}}: Point muswiki and mhwiktionary to s5 ([[phab:T259004|T259004]]; 3/3) (duration: 00m 58s)
* 08:44 urbanecm@deploy1001: Synchronized wmf-config/db-eqiad.php: {{Gerrit|81f4594b6c583f938821549b3a1800fec5b120bb}}: Point muswiki and mhwiktionary to s5 ([[phab:T259004|T259004]]; 2/3) (duration: 00m 58s)
* 08:43 urbanecm@deploy1001: Synchronized wmf-config/db-codfw.php: {{Gerrit|81f4594b6c583f938821549b3a1800fec5b120bb}}: Point muswiki and mhwiktionary to s5 ([[phab:T259004|T259004]]; 1/3) (duration: 01m 02s)
* 08:06 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|a04bc1f27e4ef4e38002d546d30bfd2d1dc60d0e}}: Turn muswiki and mhwiktionary to read-only ([[phab:T259004|T259004]]) (duration: 01m 01s)
* 08:04 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 06:54 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 06:45 XioNoX: Re-prioritize peering over transit eqiad/esams - [[phab:T259614|T259614]]
* 01:59 tstarling@deploy1001: Synchronized wmf-config/PoolCounterSettings.php: enabling fast stale mode [[phab:T250248|T250248]] (duration: 00m 58s)
* 00:33 dpifke@deploy1001: Finished deploy [performance/arc-lamp@fc5f1c6]: Deploying latest attempt to fix [[phab:T259167|T259167]] (duration: 01m 03s)
* 00:31 dpifke@deploy1001: Started deploy [performance/arc-lamp@fc5f1c6]: Deploying latest attempt to fix [[phab:T259167|T259167]]
* 00:24 mutante: reverting switch of releases.wikimedia.org for today since releases-jenkins.wikimedia.org is tied to it and new jenkins still needs some config and plugins ([[phab:T247652|T247652]])
* 00:08 mutante: releases-jenkins.wikimedia.org currently under maintenance ([[phab:T247652|T247652]])
 
== 2020-08-10 ==
* 23:56 eileen: tools revision changed from {{Gerrit|22550f38c5}} to {{Gerrit|9a89f45974}}
* 23:53 mutante: https://releases.wikimedia.org switched to new backends running Debian buster. files have been synced. httpbb tests have been created and pass. ([[phab:T247652|T247652]])
* 23:52 mutante: https://releases.wikimedia.org switched to new backends running Debian buster. files have been synced of course.
* 20:13 hashar: Updated container for Jenkins job operations-puppet-tests-buster-docker https://gerrit.wikimedia.org/r/c/integration/config/+/619359/
* 20:10 ejegg: updated payments-wiki from {{Gerrit|932aacde54}} to {{Gerrit|f199c071c3}}
* 18:32 ryankemper@deploy1001: Finished deploy [wdqs/wdqs@3e12dbb]: 0.3.44 (duration: 15m 18s)
* 18:20 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:17 volans@cumin1001: START - Cookbook sre.dns.netbox
* 18:17 ryankemper@deploy1001: Started deploy [wdqs/wdqs@3e12dbb]: 0.3.44
* 18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Special:Investigate on frwiki ([[phab:T257891|T257891]]) (duration: 00m 58s)
* 18:07 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Explicitly disable nativeGallery in Parsoid settings (no-op) (duration: 00m 58s)
* 18:04 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump the weight of near match for search ([[phab:T257922|T257922]]) (duration: 00m 59s)
* 17:56 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
* 17:54 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:52 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:50 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:49 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add eventgate-analytics streams - [[phab:T251935|T251935]] (duration: 01m 02s)
* 17:46 robh@cumin1001: START - Cookbook sre.dns.netbox
* 17:38 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 17:38 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 17:37 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:34 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 17:34 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 17:31 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:31 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 17:31 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 17:16 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:15 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 17:14 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:13 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:12 volans@cumin1001: START - Cookbook sre.dns.netbox
* 17:06 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 16:08 hnowlan@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 16:04 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:03 hnowlan@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' .
* 15:59 volans@cumin1001: START - Cookbook sre.dns.netbox
* 15:59 volans@cumin1001: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
* 15:59 volans@cumin1001: START - Cookbook sre.dns.netbox
* 15:55 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:32 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:17 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:11 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 15:02 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:01 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:48 volans@cumin1001: START - Cookbook sre.dns.netbox
* 14:16 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 14:15 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 14:15 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 14:14 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 14:14 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 14:14 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 13:55 XioNoX: Re-prioritize peering over transit - codfw - [[phab:T259614|T259614]]
* 12:34 XioNoX: Re-prioritize peering over transit - eqsin - [[phab:T259614|T259614]]
* 12:07 XioNoX: standardize cr1-eqiad interfaces
* 11:56 Urbanecm: EU B&C window done
* 11:55 Urbanecm: Run `mwscript namespaceDupes.php --wiki=tiwiki --fix` at mwmaint1002 ([[phab:T259295|T259295]])
* 11:54 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|14b2897760d2658be70765fbb8e56ad552ce7a81}}: Define Portal namespace for tiwiki ([[phab:T259295|T259295]]) (duration: 00m 59s)
* 11:49 urbanecm@deploy1001: Synchronized static/images/project-logos/: {{Gerrit|bbbf7018b1b4db1a99f4828d1907c77a19158884}}: Regenerate Bengali Wikipedia logo from source SVG ([[phab:T259292|T259292]]) (duration: 00m 59s)
* 11:41 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|0d8366f6951afae25617bea9402aa52e26f34e5d}}: Search Work NS by default at bnwikisource ([[phab:T258982|T258982]]) (duration: 00m 59s)
* 11:37 Urbanecm: Run `mwscript namespaceDupes.php --wiki=hywiki --fix` at mwmaint1002 ([[phab:T259987|T259987]])
* 11:35 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|177148777e9c11a9f936d5d8b4d1c201ba9bf7fb}}: add two extra namespaces for hywiki ([[phab:T259987|T259987]]) (duration: 00m 59s)
* 11:28 Urbanecm: Purge https://en.wikipedia.org/static/images/project-logos/shnwiktionary*.png with purgeList.php ([[phab:T260010|T260010]])
* 11:27 XioNoX: standardize cr2-eqiad interfaces
* 11:27 urbanecm@deploy1001: Synchronized static/images/project-logos/: {{Gerrit|c5c96ca419b3ab13c90cb55646be2aa9a07c8527}}: Regenerate shnwiktionary logo from source svg ([[phab:T260010|T260010]]) (duration: 00m 58s)
* 11:21 XioNoX: repool ulsfo
* 11:17 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|a15e3a22da13af89e9a2b76fdb24d6b5bebe6ec4}}: Increase autoconfirmed threshold for Chinese Wikinews to 7 days and 20 edits at least ([[phab:T259869|T259869]]) (duration: 00m 58s)
* 11:13 XioNoX: Re-prioritize peering over transit - ulsfo - [[phab:T259614|T259614]]
* 11:13 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|ba0b2abc0414940749c28a1f82ffbbfd94cd0fc5}}: Create TemplateEditor group on zhwiki ([[phab:T260012|T260012]]) (duration: 00m 58s)
* 11:09 Urbanecm: Run mwscript namespaceDupes.php --wiki=ptwikinews --fix --add-prefix=[[phab:T259959|T259959]] ([[phab:T259959|T259959]])
* 11:09 Urbanecm: Run mwscript namespaceDupes.php --wiki=ptwikinews --fix ([[phab:T259959|T259959]])
* 11:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|010f63ed64c599712e9ac11ed7fced666cc88ca1}}: Add WN as an alias to project namespace in Portuguese Wikinews ([[phab:T259959|T259959]]) (duration: 00m 58s)
* 11:06 urbanecm@deploy1001: sync-file aborted: {{Gerrit|010f63ed64c599712e9ac11ed7fced666cc88ca1}}: Add WN as an alias to project namespace in Portuguese Wikinews ([[phab:T259959|T259959]]¨) (duration: 00m 00s)
* 10:44 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:619273{{!}} Bumping portals to master (T128546)]] (duration: 00m 58s)
* 10:43 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:619273{{!}} Bumping portals to master (T128546)]] (duration: 01m 01s)
* 10:42 jayme@cumin1001: END (PASS) - Cookbook sre.discovery.pool (exit_code=0)
* 10:37 jayme@cumin1001: START - Cookbook sre.discovery.pool
* 10:36 jayme@cumin1001: END (FAIL) - Cookbook sre.discovery.pool (exit_code=99)
* 10:36 jayme@cumin1001: START - Cookbook sre.discovery.pool
* 10:33 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:32 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 10:29 jayme@cumin1001: END (PASS) - Cookbook sre.discovery.depool (exit_code=0)
* 10:26 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 10:26 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 10:23 jayme@cumin1001: START - Cookbook sre.discovery.depool
* 10:19 jayme@cumin1001: END (PASS) - Cookbook sre.discovery.pool (exit_code=0)
* 10:18 jayme@cumin1001: START - Cookbook sre.discovery.pool
* 10:14 volans@cumin1001: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:10 volans@cumin1001: START - Cookbook sre.dns.netbox
* 10:07 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
* 10:04 jayme@cumin1001: START - Cookbook sre.hosts.reboot-single
* 09:56 hashar: Updated containeer for Jenkins job operations-dns-lint-docker https://gerrit.wikimedia.org/r/619267
* 09:55 hashar: Updated container for Jenkins job operations-puppet-tests-buster-docker https://gerrit.wikimedia.org/r/619266
* 09:54 jayme@cumin1001: END (PASS) - Cookbook sre.discovery.depool (exit_code=0)
* 09:49 jayme@cumin1001: START - Cookbook sre.discovery.depool
* 09:21 marostegui: Promote dbproxy1019 back [[phab:T255408|T255408]]
* 08:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 08:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 06:43 marostegui: Remove revision triggers from db2094:3318 [[phab:T238966|T238966]]
* 06:42 marostegui: Stop replication on s8 codfw master to deploy MCR change, this will generate lag on s8 codfw [[phab:T238966|T238966]]
* 04:46 marostegui: Depool dbproxy1019 for reimage [[phab:T255408|T255408]]
 
== 2020-08-09 ==
* 21:58 ejegg: updated payments-wiki from {{Gerrit|cd012f37f1}} to {{Gerrit|932aacde54}}
* 03:53 ryankemper@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
 
== 2020-08-08 ==
* 02:23 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload
* 02:21 ryankemper@cumin1001: END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97)
* 02:19 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload
 
== 2020-08-07 ==
* 16:42 jforrester@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/DiscussionTools/: [[phab:T259855|T259855]] Revert new reply API (duration: 01m 06s)
* 15:01 volans: import DNS names for network devices in Netbox - [[phab:T258729|T258729]]
* 13:27 godog: bounce pybal on lvs1016 and then lvs1015 to reset state, logstash1025 reported down but actually up
* 10:27 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 10:27 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
* 10:02 elukey: reboot deneb via ganeti2021 (hostname config pointing to recdns for some reason)
* 09:15 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1092', diff saved to https://phabricator.wikimedia.org/P12195 and previous config saved to /var/cache/conftool/dbconfig/20200807-091527-marostegui.json
* 08:47 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12194 and previous config saved to /var/cache/conftool/dbconfig/20200807-084747-marostegui.json
* 08:07 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12193 and previous config saved to /var/cache/conftool/dbconfig/20200807-080719-marostegui.json
* 07:50 godog: prometheus codfw lvextend --resize --size +60G /dev/mapper/vg--hdd-prometheus--global
* 07:49 godog: prometheus codfw lvextend --resize --size +30G /dev/mapper/vg--ssd-prometheus--k8s
* 07:46 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12192 and previous config saved to /var/cache/conftool/dbconfig/20200807-074658-marostegui.json
* 06:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 06:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 06:34 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1092 for upgrade', diff saved to https://phabricator.wikimedia.org/P12191 and previous config saved to /var/cache/conftool/dbconfig/20200807-063431-marostegui.json
 
== 2020-08-06 ==
* 23:21 catrope@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/: Fixes for WelcomeSurvey language question ([[phab:T232410|T232410]]) (duration: 00m 59s)
* 23:04 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Change GrowthExperiments mentor list on fawiki ([[phab:T253291|T253291]]) (duration: 00m 59s)
* 21:43 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 21:41 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 21:40 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 21:39 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
* 21:39 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 21:35 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' .
* 21:33 brennen@deploy1001: Synchronized php-1.36.0-wmf.3/vendor: [[gerrit:618850{{!}}Update git submodules (vendor)]] ([[phab:T259832|T259832]]) (duration: 01m 08s)
* 21:32 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' .
* 20:51 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 20:51 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 20:47 shdubsh: restart logstash -- pipeline appears stuck
* 20:38 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 20:38 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 20:19 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 20:19 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 20:19 brennen: manually updating the vendor submodule on 1.36.0 for [[phab:T259832|T259832]]
* 20:15 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 20:15 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 19:48 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 19:47 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 19:45 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - wgEventStreams - fix another typo in eventgate stream config - [[phab:T251935|T251935]] (duration: 00m 58s)
* 19:40 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - wgEventStreams - fix typo in eventgate stream config - [[phab:T251935|T251935]] (duration: 00m 59s)
* 19:26 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 19:26 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 19:04 brennen@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.3
* 18:58 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' .
* 18:57 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' .
* 18:29 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 18:29 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 18:21 Urbanecm: Morning B&C window was completed
* 18:20 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/modules/: {{Gerrit|fb4a80830d7d915479e097cc82c681c5fb03d51b}}: Fix "Ask mentor" help panel button styling ([[phab:T250235|T250235]]) (duration: 01m 07s)
* 18:11 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 18:11 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 18:10 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|9db96595695b5ec1144c078e8961b3c04e8983cf}}: Remove temporary logging for mediamoderation ([[phab:T259742|T259742]]) (duration: 01m 07s)
* 18:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|9695811a30de30471a81b6ad05aa5e625f52caf1}}: : Enable DiscussionTools as a beta feature on 8 more wikis ("phase 1") ([[phab:T259574|T259574]]) (duration: 01m 06s)
* 17:42 brennen@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.3 (duration: 01m 06s)
* 17:41 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.3
* 17:37 brennen: train 1.36.0-wmf.3: proceeding to group1
* 17:36 brennen@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/WikibaseMediaInfo/src/View/MediaInfoEntityTermsView.php: Backport: [[gerrit:618582{{!}}Fix array unpacking as argument list]] ([[phab:T259745|T259745]]) (duration: 01m 07s)
* 16:32 chrisalbon@deploy1001: Finished deploy [ores/deploy@f3c44be]: [[phab:T258435|T258435]] (duration: 14m 12s)
* 16:18 dpifke@deploy1001: Finished deploy [performance/arc-lamp@7838c88]: Deploying fixes for [[phab:T259167|T259167]] (duration: 00m 05s)
* 16:18 dpifke@deploy1001: Started deploy [performance/arc-lamp@7838c88]: Deploying fixes for [[phab:T259167|T259167]]
* 16:18 chrisalbon@deploy1001: Started deploy [ores/deploy@f3c44be]: [[phab:T258435|T258435]]
* 15:40 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' .
* 15:40 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' .
* 15:10 fdans@deploy1001: Finished deploy [analytics/refinery@97a02a3]: Regular analytics weekly train [analytics/refinery@97a02a3 (duration: 20m 01s)
* 14:50 fdans@deploy1001: Started deploy [analytics/refinery@97a02a3]: Regular analytics weekly train [analytics/refinery@97a02a3
* 14:00 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add eventgate-* test.event streams - [[phab:T251935|T251935]] (duration: 01m 08s)
* 13:32 jayme: updated helm to 2.16.9-2 on contint*, deploy* and chartmuseum*
* 13:24 jayme: imported helm_2.16.9-2 and tiller_2.16.9-2 to buster-wikimedia, jessie-wikimedia and stretch-wikimedia
* 12:06 kart_: Updated cxserver to 2020-08-05-070016-production ([[phab:T258919|T258919]], [[phab:T199523|T199523]], [[phab:T257943|T257943]], [[phab:T256194|T256194]])
* 12:03 kartik@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' .
* 11:59 kartik@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' .
* 11:57 kartik@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
* 11:54 Lucas_WMDE: EU backport window done
* 11:54 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/Flow/: Backport: [[gerrit:618580{{!}}Pass jQuery objects into jqueryMsg]] (duration: 01m 09s)
* 11:53 XioNoX: reboot cr2-eqord - [[phab:T259621|T259621]]
* 11:37 XioNoX: drain traffic away cr2-eqord - [[phab:T259621|T259621]]
* 11:27 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/Wikibase/lib/: Backport: [[gerrit:618579{{!}}Fix CachingFallbackLabelDescriptionLookup failing in edge-cases (T259744)]] (duration: 01m 10s)
* 11:22 XioNoX: reboot cr2-eqdfw - [[phab:T259621|T259621]]
* 11:13 XioNoX: drain traffic away cr2-eqdfw - [[phab:T259621|T259621]]
* 10:52 mvolz@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'zotero' for release 'production' .
* 10:48 mvolz@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'zotero' for release 'production' .
* 10:45 mvolz@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' .
* 10:23 mvolz@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' .
* 10:16 mvolz@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' .
* 10:14 jynus@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 10:12 jynus@cumin2001: START - Cookbook sre.hosts.downtime
* 10:11 mvolz@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' .
* 08:44 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1127', diff saved to https://phabricator.wikimedia.org/P12188 and previous config saved to /var/cache/conftool/dbconfig/20200806-084406-marostegui.json
* 08:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1127', diff saved to https://phabricator.wikimedia.org/P12187 and previous config saved to /var/cache/conftool/dbconfig/20200806-083743-marostegui.json
* 08:30 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1127', diff saved to https://phabricator.wikimedia.org/P12186 and previous config saved to /var/cache/conftool/dbconfig/20200806-083033-marostegui.json
* 08:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1127', diff saved to https://phabricator.wikimedia.org/P12185 and previous config saved to /var/cache/conftool/dbconfig/20200806-081416-marostegui.json
* 07:03 elukey@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0)
* 06:57 elukey@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper
* 06:57 marostegui: Truncate tables on zerowiki [[phab:T227717|T227717]]
* 06:53 elukey@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0)
* 06:47 elukey@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper
* 06:43 elukey@cumin1001: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0)
* 06:37 elukey: roll restart of druid clusters' zookeeper and an-conf* zookeeper for openjdk-11 upgrades
* 06:36 elukey@cumin1001: START - Cookbook sre.zookeeper.roll-restart-zookeeper
* 06:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 06:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 05:07 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1127 for MCR', diff saved to https://phabricator.wikimedia.org/P12184 and previous config saved to /var/cache/conftool/dbconfig/20200806-050743-marostegui.json
* 04:56 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1079', diff saved to https://phabricator.wikimedia.org/P12182 and previous config saved to /var/cache/conftool/dbconfig/20200806-045622-marostegui.json
* 04:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079', diff saved to https://phabricator.wikimedia.org/P12181 and previous config saved to /var/cache/conftool/dbconfig/20200806-045107-marostegui.json
* 04:46 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079', diff saved to https://phabricator.wikimedia.org/P12180 and previous config saved to /var/cache/conftool/dbconfig/20200806-044608-marostegui.json
* 04:37 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1079', diff saved to https://phabricator.wikimedia.org/P12179 and previous config saved to /var/cache/conftool/dbconfig/20200806-043758-marostegui.json
* 03:04 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp2019.codfw.wmnet
* 02:24 eileen: process-control config revision is {{Gerrit|525eb71235}} turn off delete deleted contacts
* 01:52 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 01:52 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 01:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 01:19 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 01:17 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 01:17 dzahn@cumin1001: START - Cookbook sre.hosts.downtime
* 00:35 mutante: wtp2019 - reimaging - parsoid service does not work, unlike on all other wtp*, making sure it's clean
* 00:00 mutante: LDAP - removed demon from nda group
 
== 2020-08-05 ==
* 23:57 eileen: civicrm revision changed from {{Gerrit|150c3476c4}} to {{Gerrit|72452e28a9}}, config revision is {{Gerrit|b6ece03513}}
* 23:02 shdubsh: logstash in codfw looks stuck -- restarting
* 19:41 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert group1 wikis to 1.36.0-wmf.2
* 19:39 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 19:37 pt1979@cumin2001: START - Cookbook sre.hosts.downtime
* 19:13 brennen@deploy1001: Synchronized php: group1 wikis to 1.36.0-wmf.3 (duration: 01m 44s)
* 19:11 brennen@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.3
* 18:26 Lucas_WMDE: Morning backport window done
* 18:25 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/ContentTranslation/: Backport: [[gerrit:618566{{!}}Pass jQuery objects into jqueryMsg]] (duration: 01m 11s)
* 18:14 mutante: test !log
* 18:11 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:618343{{!}}Re-enable growth study quick survey (T257015)]] (duration: 01m 12s)
* 17:30 shdubsh: test prometheus-icinga-exporter upgrade on icinga2001
* 16:50 elukey: powercycle stat1005 after GPU issue
* 15:56 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add eventgate-logging-external streams and destination_event_service settings - [[phab:T251935|T251935]] (duration: 01m 05s)
* 15:50 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:43 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:11 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:08 godog: bounce logstash on logstash100[789] - udp loss reported
* 15:05 pt1979@cumin2001: START - Cookbook sre.dns.netbox
* 14:48 elukey: reboot stat1008 for unexpected maintenance (GPU stuck)
* 14:33 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 14:32 otto@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 14:27 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 14:27 otto@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 14:25 moritzm: installing nmap bugfix updates from buster point release
* 14:24 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
* 14:24 otto@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
* 14:20 sukhe@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 14:20 sukhe@cumin1001: START - Cookbook sre.hosts.downtime
* 14:14 moritzm: installing pillow security updates
* 14:03 moritzm: installing node-minimist security updates
* 13:51 moritzm: installing Linux update to 4.9.132 from buster point update (no reboots, just the package updates)
* 13:32 jayme: updated helmfile to 0.125.2-0 and helm-diff to 3.1.2-1 on contint* and deploy*
* 13:28 volans@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:24 volans@cumin1001: START - Cookbook sre.dns.netbox
* 13:04 elukey: restart yarn resource managers on an-master100[12] to pick up new Yarn settings - https://gerrit.wikimedia.org/r/c/operations/puppet/+/618529
* 13:00 moritzm: installing libjpeg-turbo security updates on stretch
* 12:52 XioNoX: netmon1002:/srv/deployment/librenms/librenms$ sudo -u librenms ./lnms migrate
* 12:49 jayme: imported helm-diff_3.1.2-1 to buster-wikimedia, jessie-wikimedia and stretch-wikimedia
* 12:46 moritzm: installing imagemagick security updates on buster
* 12:33 moritzm: installing net-snmp security updates on icinga hosts
* 11:36 awight: EU Bacon reclosed
* 11:36 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:614891{{!}}Switch test wikis to new version of vector by default (3/3) (T254227)]] (duration: 01m 07s)
* 11:29 awight: EU Bacon reopened
* 11:28 awight: EU Bacon complete
* 11:26 awight@deploy1001: Synchronized wmf-config: Config: [[gerrit:618481{{!}}FileImporter: full default deployment (T232542)]] (duration: 01m 04s)
* 11:23 jayme: imported helm-diff_3.1.2-0 to jessie-wikimedia and stretch-wikimedia
* 11:22 jayme: imported helm-diff_3.1.2-0 to buster-wikimedia
* 11:19 awight@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:618303{{!}}Add import sources for lijwikisource (T259633)]] (duration: 01m 07s)
* 11:13 awight@deploy1001: sync-file aborted: Config: [[gerrit:618303{{!}}Add import sources for lijwikisource (T259633)]] (duration: 00m 13s)
* 11:10 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:595542{{!}}Enable Data Bridge on Test Wikidata clients (T232584)]] (duration: 01m 20s)
* 10:39 XioNoX: reboot cr3-ulsfo - [[phab:T259621|T259621]]
* 10:28 XioNoX: drain traffic away cr3-ulsfo - [[phab:T259621|T259621]]
* 10:21 moritzm: installing libssh security updates
* 10:18 XioNoX: reboot cr4-ulsfo - [[phab:T259621|T259621]]
* 09:58 XioNoX: drain traffic away cr4-ulsfo
* 09:53 XioNoX: depool ulsfo - [[phab:T259621|T259621]]
* 09:32 elukey: set ticket max renewable lifetime to 7d on all kerberos clients (was zero, the default)
* 09:07 jayme: imported helmfile_0.125.2-0 to jessie-wikimedia
* 09:07 jayme: imported helmfile_0.125.2-0 to stretch-wikimedia
* 09:05 jayme: imported helmfile_0.125.2-0 to buster-wikimedia
* 08:39 marostegui: Remove revision triggers on db1125:3317
* 08:39 marostegui: Stop replication on db1079 for MCR, this will generate lag on s7 on labsdb
* 08:39 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1079 for MCR', diff saved to https://phabricator.wikimedia.org/P12173 and previous config saved to /var/cache/conftool/dbconfig/20200805-083916-marostegui.json
* 08:38 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P12172 and previous config saved to /var/cache/conftool/dbconfig/20200805-083833-marostegui.json
* 08:29 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P12171 and previous config saved to /var/cache/conftool/dbconfig/20200805-082908-marostegui.json
* 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P12170 and previous config saved to /var/cache/conftool/dbconfig/20200805-082138-marostegui.json
* 08:12 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P12169 and previous config saved to /var/cache/conftool/dbconfig/20200805-081237-marostegui.json
* 07:49 marostegui: Stop mysql on db1117:3323 (this will generate haproxy irc alerts) [[phab:T259589|T259589]]
* 07:45 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 07:39 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 07:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 07:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 07:26 moritzm: installing perl security updates on buster
* 07:20 moritzm: installing libexif security updates on buster
* 07:14 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 07:13 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 07:04 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 07:04 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 06:59 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 06:59 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 06:52 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 06:50 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 06:50 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 06:46 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 06:46 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 05:53 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 05:09 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1094 for MCR', diff saved to https://phabricator.wikimedia.org/P12167 and previous config saved to /var/cache/conftool/dbconfig/20200805-050907-marostegui.json
* 05:08 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1136', diff saved to https://phabricator.wikimedia.org/P12166 and previous config saved to /var/cache/conftool/dbconfig/20200805-050808-marostegui.json
* 05:03 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1136', diff saved to https://phabricator.wikimedia.org/P12165 and previous config saved to /var/cache/conftool/dbconfig/20200805-050308-marostegui.json
* 04:53 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1136', diff saved to https://phabricator.wikimedia.org/P12164 and previous config saved to /var/cache/conftool/dbconfig/20200805-045334-marostegui.json
* 04:33 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1136', diff saved to https://phabricator.wikimedia.org/P12163 and previous config saved to /var/cache/conftool/dbconfig/20200805-043346-marostegui.json
 
== 2020-08-04 ==
* 22:41 brennen: restarting php7.2-fpm on mw1404 for opcache issues
* 21:45 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' .
* 21:38 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' .
* 21:34 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' .
* 21:03 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
* 21:03 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
* 20:55 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' .
* 20:55 mholloway-shell@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
* 20:52 mholloway-shell@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' .
* 20:27 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@c80e2e7]: use provided ca certs for elasticsearch (duration: 02m 22s)
* 20:25 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@c80e2e7]: use provided ca certs for elasticsearch
* 20:15 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@b17bfd4]: Move mjolnir daemons from cirrus hosts to dedicated instances (duration: 02m 07s)
* 20:12 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@b17bfd4]: Move mjolnir daemons from cirrus hosts to dedicated instances
* 19:19 brennen@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.3
* 19:11 brennen@deploy1001: Finished scap: testwikis wikis to 1.36.0-wmf.3 (duration: 91m 03s)
* 19:03 brennen: current 1.36.0-wmf.3 train status ([[phab:T257971|T257971]]): mid scap-cdb-rebuild for testwiki sync; will proceed with group0 when finished.
* 18:55 sukhe: upload pdns-recursor_4.3.3-1~deb10u1 to apt.wm.o (buster) - [[phab:T252132|T252132]]
* 18:49 mutante: letting puppet install envoy on all ores1* hosts
* 18:46 mutante: letting puppet install envoy on all ores2* hosts
* 18:37 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 18:26 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 18:19 mutante: temp disabling puppet on all ores hosts to add envoy
* 17:50 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:40 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:40 brennen@deploy1001: Started scap: testwikis wikis to 1.36.0-wmf.3
* 17:36 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:25 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:21 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:17 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 17:05 brennen: 1.36.0-wmf.3 was branched at {{Gerrit|2d0cf09cdf}} for [[phab:T257971|T257971]]
* 16:51 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:49 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:43 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:31 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:24 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 16:18 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:15 oblivian@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' .
* 16:08 cmjohnson@cumin1001: START - Cookbook sre.dns.netbox
* 16:05 oblivian@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' .
* 16:02 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 16:02 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 15:53 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Set default topic_prefixes - [[phab:T255888|T255888]] (duration: 00m 58s)
* 15:51 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0)
* 15:39 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' .
* 15:39 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' .
* 15:38 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:31 hnowlan@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' .
* 15:18 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove now unused wgEventServiceStreamConfig - [[phab:T229863|T229863]] (duration: 00m 58s)
* 15:18 moritzm: installing jackson-databind security issues
* 15:08 moritzm: installing qemu security updates on cloudvirt* Stretch hosts
* 14:54 cmjohnson1: swapping kubernetes1010 network cable [[phab:T257542|T257542]]
* 14:48 ayounsi@cumin1001: START - Cookbook sre.network.prepare-upgrade
* 14:41 cmjohnson1: powercycling analytics1050 [[phab:T258370|T258370]]
* 14:35 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1136 for MCR', diff saved to https://phabricator.wikimedia.org/P12161 and previous config saved to /var/cache/conftool/dbconfig/20200804-143524-marostegui.json
* 14:27 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P12160 and previous config saved to /var/cache/conftool/dbconfig/20200804-142710-marostegui.json
* 14:22 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P12159 and previous config saved to /var/cache/conftool/dbconfig/20200804-142220-marostegui.json
* 14:15 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P12158 and previous config saved to /var/cache/conftool/dbconfig/20200804-141556-marostegui.json
* 14:10 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P12157 and previous config saved to /var/cache/conftool/dbconfig/20200804-141004-marostegui.json
* 13:56 akosiaris@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
* 13:56 akosiaris@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
* 13:56 akosiaris@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' .
* 13:51 hashar: Install newer openjdk on contint2001 and restarting CI Jenkins
* 12:00 jayme: helm was updated: 2.16.7-2 -> 2.16.9-1 on chartmuseum*, contint*, deploy*
* 11:43 Lucas_WMDE: EU backport window done
* 11:41 marostegui: Deploy schema change on s3 codfw master, lag might show up on codfw s3 [[phab:T259238|T259238]]
* 11:37 moritzm: installing openjdk-11 security updates
* 11:36 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/Wikibase.php: Config: [[gerrit:618266{{!}}Load WikibaseRepo using extension registration in production (T257433)]] (duration: 00m 58s)
* 11:12 Lucas_WMDE: Deployed patch for [[phab:T86738|T86738]] / [[phab:T259565|T259565]]
* 11:03 moritzm: installing e2fsprogs security updates for stretch
* 10:47 moritzm: installing tomcat8 security updates
* 10:47 vgutierrez: upgrade acme-chief to version 0.28
* 10:33 vgutierrez: upload acme-chief 0.28 to apt.wm.o (buster) - [[phab:T259338|T259338]]
* 10:18 moritzm: installing imagemagick security updates on stretch
* 10:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1101:3317 for MCR and PK change [[phab:T259524|T259524]]', diff saved to https://phabricator.wikimedia.org/P12156 and previous config saved to /var/cache/conftool/dbconfig/20200804-100035-marostegui.json
* 09:56 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P12155 and previous config saved to /var/cache/conftool/dbconfig/20200804-095608-marostegui.json
* 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1098:3317', diff saved to https://phabricator.wikimedia.org/P12154 and previous config saved to /var/cache/conftool/dbconfig/20200804-094909-marostegui.json
* 09:06 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 09:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 08:58 moritzm: installing python3.5 security updates
* 08:15 moritzm: installing remaining cups security updates
* 08:13 XioNoX: cleaning up a bunch of prefix limit reached issues
* 08:00 marostegui: Failover m2 from db1132 to db1107 -[[phab:T257540|T257540]]
* 07:54 moritzm: installing poppler security updates on stretch
* 07:43 jayme: imported helm_2.16.9-1 to jessie-wikimedia
* 07:43 jayme: imported helm_2.16.9-1 to stretch-wikimedia
* 07:38 jayme: imported helm_2.16.9-1 to buster-wikimedia
* 07:34 elukey: upgrade druid analytics (backend for Turnilo/Superset/etc..) to 0.19
* 07:32 XioNoX: remove nonstop-bridging from fasw-c-eqiad switches - [[phab:T191667|T191667]]
* 07:29 XioNoX: remove nonstop-bridging from eqiad asw2 switches - [[phab:T191667|T191667]]
* 07:28 XioNoX: remove nonstop-bridging from asw2-esams - [[phab:T191667|T191667]]
* 07:27 marostegui: Start topology changes on m2 - [[phab:T257540|T257540]]
* 07:25 moritzm: installing rails security updates
* 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1119', diff saved to https://phabricator.wikimedia.org/P12153 and previous config saved to /var/cache/conftool/dbconfig/20200804-064223-marostegui.json
* 06:30 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P12152 and previous config saved to /var/cache/conftool/dbconfig/20200804-063026-marostegui.json
* 06:27 _joe_: restarting docker daemon on kubestage1002, seems like a case of https://github.com/moby/moby/issues/29635
* 06:23 marostegui@cumin1001: dbctl commit (dc=all): 'Restore original weight to db1089 on main traffic', diff saved to https://phabricator.wikimedia.org/P12151 and previous config saved to /var/cache/conftool/dbconfig/20200804-062358-marostegui.json
* 06:22 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P12150 and previous config saved to /var/cache/conftool/dbconfig/20200804-062256-marostegui.json
* 06:19 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
* 06:13 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: re-enabling lilypond execution in safe mode 3rd attempt (duration: 00m 58s)
* 06:12 marostegui@cumin1001: dbctl commit (dc=all): 'More weight to db1089 on main traffic', diff saved to https://phabricator.wikimedia.org/P12149 and previous config saved to /var/cache/conftool/dbconfig/20200804-061255-marostegui.json
* 06:12 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P12148 and previous config saved to /var/cache/conftool/dbconfig/20200804-061209-marostegui.json
* 06:10 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1098:3317 for MCR', diff saved to https://phabricator.wikimedia.org/P12147 and previous config saved to /var/cache/conftool/dbconfig/20200804-061003-marostegui.json
* 05:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 05:35 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 05:18 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1119 for reimage', diff saved to https://phabricator.wikimedia.org/P12146 and previous config saved to /var/cache/conftool/dbconfig/20200804-051843-marostegui.json
* 05:04 marostegui: Reboot db1107 to pick up the last kernel
* 05:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1089 into API', diff saved to https://phabricator.wikimedia.org/P12145 and previous config saved to /var/cache/conftool/dbconfig/20200804-050150-marostegui.json
* 03:56 legoktm: added Arlo to wmf-deployment Gerrit group
* 03:53 legoktm: added subbu to wmf-deployment Gerrit group
 
== 2020-08-03 ==
* 23:43 mutante: mwdebug1001 - temp installing apt-file for debugging an issue on mwmaint
* 23:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable GrowthExperiments on fawiki ([[phab:T253291|T253291]]) (duration: 00m 59s)
* 21:35 sbassett: Deployed mitigations for [[phab:T115888|T115888]]
* 21:14 sbassett@deploy1001: Synchronized php-1.36.0-wmf.2/resources/src/mediawiki.jqueryMsg/mediawiki.jqueryMsg.js: (no justification provided) (duration: 01m 00s)
* 18:15 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 18:13 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 18:13 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 18:13 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 18:09 dcausse@deploy1001: Finished deploy [wdqs/wdqs@20dcff3]: deploy 0.3.43 and gui update (duration: 15m 53s)
* 17:53 dcausse@deploy1001: Started deploy [wdqs/wdqs@20dcff3]: deploy 0.3.43 and gui update
* 17:33 liw@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.2
* 17:28 dcausse@deploy1001: Finished deploy [wdqs/wdqs@20dcff3]: (no justification provided) (duration: 00m 35s)
* 17:28 dcausse@deploy1001: Started deploy [wdqs/wdqs@20dcff3]: (no justification provided)
* 16:58 liw@deploy1001: rebuilt and synchronized wikiversions files: Revert "group2 wikis to 1.36.0-wmf.1"
* 16:21 oblivian@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 16:16 oblivian@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
* 16:02 oblivian@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
* 15:55 _joe_: regenerating the TLS certs for blubberoid
* 15:33 XioNoX: standardize all routers routing-options config
* 15:27 marostegui: Change PK on frwiktionary.revision on db2087:3317, db2129, db2121 db2086:3317 [[phab:T259524|T259524]]
* 15:16 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:14 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 15:12 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:12 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:12 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:12 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:12 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
* 15:12 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:11 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 15:10 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 14:51 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1106', diff saved to https://phabricator.wikimedia.org/P12143 and previous config saved to /var/cache/conftool/dbconfig/20200803-145111-marostegui.json
* 14:40 moritzm: update Buster netboot images to Buster 10.5 [[phab:T259519|T259519]]
* 14:33 XioNoX: disable all ALGs from pfw3-codfw
* 14:28 XioNoX: remove IGMP and PIM from pfw3-codfw security zones
* 14:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1089 into dump and depool db1106', diff saved to https://phabricator.wikimedia.org/P12142 and previous config saved to /var/cache/conftool/dbconfig/20200803-142749-marostegui.json
* 14:27 XioNoX: remove nonstop-bridging from fasw-c-codfw - [[phab:T191667|T191667]]
* 14:22 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 14:20 andrew@cumin1001: START - Cookbook sre.hosts.downtime
* 14:04 filippo@deploy1001: Finished deploy [librenms/librenms@413e006]: Upgrade LibreNMS to 1.66 - [[phab:T257017|T257017]] (duration: 00m 23s)
* 14:03 filippo@deploy1001: Started deploy [librenms/librenms@413e006]: Upgrade LibreNMS to 1.66 - [[phab:T257017|T257017]]
* 14:00 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕙☕ sudo cumin A:puppetmaster 'enable-puppet "cdanis deploying I92e9a05"'
* 13:56 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕙☕ sudo cumin A:puppetmaster 'disable-puppet "cdanis deploying I92e9a05"'
* 13:27 moritzm: installing libopenmpt security updates
* 13:15 XioNoX: remove nonstop-bridging from asw-d-codfw - [[phab:T191667|T191667]]
* 13:14 XioNoX: remove nonstop-bridging from asw-c-codfw - [[phab:T191667|T191667]]
* 13:12 XioNoX: remove nonstop-bridging from asw-b-codfw - [[phab:T191667|T191667]]
* 13:11 XioNoX: remove nonstop-bridging from asw-a-codfw - [[phab:T191667|T191667]]
* 13:05 moritzm: installing json-c security updates
* 12:53 XioNoX: move VRRP master to cr3-eqsin
* 12:32 liw@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.2
* 12:26 moritzm: installing apache-log4j1.2 security updates
* 12:20 moritzm: restarting nginx on francium to pick up luajit update
* 12:13 kormat: disabling puppet on cumin hosts [[phab:T259021|T259021]]
* 11:55 moritzm: installing luajit security updates
* 11:20 moritzm: installing ruby-rack security updates
* 11:19 Urbanecm: EU B&C done
* 11:19 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|346138d95721c274d450568388fb2ad1803dba9e}}: Add extra namespaces for yuewiktionary ([[phab:T258913|T258913]]) (duration: 01m 06s)
* 11:12 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|8c2a2b2187fcc33c37b9d0f6bafcc963afd0b74b}}: Add gpophotoeng.gov.il to the wgCopyUploadsDomains allowlist for commonswiki ([[phab:T258857|T258857]]) (duration: 01m 07s)
* 11:03 urbanecm@deploy1001: Synchronized wmf-config/throttle.php: {{Gerrit|ead6b9eb699594583b06b8f5c23d40d9add2eb49}}: New throttle rule for Czech editathon ([[phab:T259352|T259352]]) (duration: 01m 06s)
* 11:03 moritzm: installing ruby2.5 security updates
* 11:01 moritzm: removing cloudcephmon100[1-3].wikimedia.org from debmonitor (these eventually got re-installed as cloudcephmon100[1-3].eqiad.wmnet)
* 10:51 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:618020{{!}} Bumping portals to master (T128546)]] (duration: 01m 06s)
* 10:50 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:618020{{!}} Bumping portals to master (T128546)]] (duration: 01m 08s)
* 10:29 moritzm: installing NSS security updates on buster
* 10:26 moritzm: restarting Apache on puppetboard to pick up curl security updates
* 10:19 moritzm: restarting wtp1025 (parsoid canary) to pick up curl security updates
* 09:46 moritzm: restarting mw1261-mw1265 to pick up curl security updates
* 09:42 moritzm: installing curl security updates on stretch
* 08:59 moritzm: installing ffmpeg security updates on jobrunners/video scalers (3.2.15 rebuilt with VP9/row-mt patches)
* 08:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db1089 into API', diff saved to https://phabricator.wikimedia.org/P12141 and previous config saved to /var/cache/conftool/dbconfig/20200803-082641-marostegui.json
* 08:25 moritzm: installing qemu security updates on stretch
* 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1106 after compression', diff saved to https://phabricator.wikimedia.org/P12140 and previous config saved to /var/cache/conftool/dbconfig/20200803-082533-marostegui.json
* 08:22 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Clarify s5 wikis [[phab:T259437|T259437]] (duration: 01m 05s)
* 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Clarify s5 wikis [[phab:T259437|T259437]] (duration: 01m 40s)
* 08:07 elukey: roll restart aqs on aqs* to pick up new druid settings
* 07:10 marostegui: Remove revision triggers from db2095:3317 for MCR changes [[phab:T238966|T238966]]
* 07:09 marostegui: Deploy MCR change on s7 codfw, lag will appear on codfw [[phab:T238966|T238966]]
* 07:07 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1106 after compression', diff saved to https://phabricator.wikimedia.org/P12139 and previous config saved to /var/cache/conftool/dbconfig/20200803-070702-marostegui.json
* 05:27 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1106 after compression', diff saved to https://phabricator.wikimedia.org/P12138 and previous config saved to /var/cache/conftool/dbconfig/20200803-052715-marostegui.json
* 05:04 marostegui: Remove db1108:3321 and db1108:3322 from tendril and add db1108:3351 and db1108:3352 [[phab:T254462|T254462]]
* 05:01 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1106 after compression', diff saved to https://phabricator.wikimedia.org/P12137 and previous config saved to /var/cache/conftool/dbconfig/20200803-050148-marostegui.json
 
== 2020-08-01 ==
* 16:30 Amir1: wikiadmin@10.64.32.197(avkwiki)> delete from site_identifiers; ([[phab:T259122|T259122]])
* 16:27 Amir1: start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https ([[phab:T259122|T259122]])
 
==Archives==
See [[Server Admin Log/Archives]].
See [[Server Admin Log/Archives]].
<noinclude>
<noinclude>

Latest revision as of 21:05, 3 February 2023

2023-02-03

  • 21:05 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:04 cmooney@cumin1001: START - Cookbook sre.dns.netbox
  • 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
  • 21:02 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
  • 21:00 cmooney@cumin1001: START - Cookbook sre.dns.netbox
  • 20:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 20:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 19:44 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet
  • 19:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1090.eqiad.wmnet with OS bullseye
  • 19:00 dzahn@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "test what is not synced - dzahn@cumin2002"
  • 18:59 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test what is not synced - dzahn@cumin2002"
  • 18:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage
  • 18:49 topranks: Enabling 4x10G channelization for pic 0 QSFP 4 on cr1-codfw
  • 18:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage
  • 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1090.eqiad.wmnet with OS bullseye
  • 18:23 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet
  • 18:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS bullseye
  • 17:57 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage
  • 17:57 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage
  • 17:39 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet
  • 17:36 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1089.eqiad.wmnet with OS bullseye
  • 17:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1088.eqiad.wmnet with OS bullseye
  • 17:34 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet
  • 17:34 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1086.eqiad.wmnet with OS bullseye
  • 17:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage
  • 17:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage
  • 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage
  • 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage
  • 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1086.eqiad.wmnet with OS bullseye
  • 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1089.eqiad.wmnet with OS bullseye
  • 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
  • 16:44 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001"
  • 16:41 cmooney@cumin1001: START - Cookbook sre.dns.netbox
  • 16:32 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2012.codfw.wmnet
  • 16:25 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2012.codfw.wmnet
  • 15:51 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): test (duration: 00m 26s)
  • 15:51 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): test
  • 15:23 milimetric@deploy1002: Finished deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection (duration: 00m 15s)
  • 15:22 milimetric@deploy1002: Started deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection
  • 14:31 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 09s)
  • 14:31 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
  • 14:20 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2011.codfw.wmnet
  • 14:19 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 23s)
  • 14:18 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
  • 14:13 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2011.codfw.wmnet
  • 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=ats-be
  • 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=cdn
  • 13:51 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1087.eqiad.wmnet with OS bullseye
  • 13:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage
  • 13:25 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage
  • 13:05 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1087.eqiad.wmnet with OS bullseye
  • 12:09 moritzm: installing node-moment security updates
  • 12:01 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 13s)
  • 12:00 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided)
  • 11:58 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2010.codfw.wmnet
  • 11:58 moritzm: installing node-qs security updates
  • 11:50 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2010.codfw.wmnet
  • 11:35 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2009.codfw.wmnet
  • 11:28 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2009.codfw.wmnet
  • 10:44 moritzm: updating perf on buster hosts
  • 10:24 stevemunene@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 10:11 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 10:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2008.codfw.wmnet
  • 10:07 stevemunene@cumin1001: END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 10:06 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 10:03 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2008.codfw.wmnet
  • 09:51 moritzm: installing ruby-rack security updates
  • 09:31 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:31 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:24 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:23 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:23 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:19 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet
  • 09:14 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet
  • 09:13 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:13 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:07 moritzm: installing modsecurity-crs security updates
  • 09:02 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet
  • 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet
  • 05:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1084.eqiad.wmnet with OS bullseye
  • 05:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1085.eqiad.wmnet with OS bullseye
  • 04:50 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage
  • 04:47 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage
  • 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage
  • 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage
  • 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1084.eqiad.wmnet with OS bullseye
  • 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1085.eqiad.wmnet with OS bullseye
  • 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet
  • 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet
  • 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1083.eqiad.wmnet with OS bullseye
  • 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1082.eqiad.wmnet with OS bullseye
  • 03:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage
  • 03:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage
  • 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage
  • 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage
  • 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1082.eqiad.wmnet with OS bullseye
  • 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1083.eqiad.wmnet with OS bullseye
  • 03:20 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet
  • 03:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1080.eqiad.wmnet with OS bullseye
  • 02:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage
  • 02:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage
  • 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=ats-be
  • 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=cdn
  • 02:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1081.eqiad.wmnet with OS bullseye
  • 02:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye
  • 02:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage
  • 02:00 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage
  • 01:38 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS bullseye
  • 01:31 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye
  • 00:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye

2023-02-02

  • 22:58 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye
  • 22:15 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet
  • 22:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS bullseye
  • 22:01 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye
  • 22:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet
  • 21:58 zabe@deploy1002: Finished scap: Backport for Stop writing to cuc_comment everywhere (T233004) (duration: 07m 58s)
  • 21:52 zabe@deploy1002: zabe: Backport for Stop writing to cuc_comment everywhere (T233004) synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
  • 21:50 zabe@deploy1002: Started scap: Backport for Stop writing to cuc_comment everywhere (T233004)
  • 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1078.eqiad.wmnet with OS bullseye
  • 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage
  • 21:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage
  • 21:30 brennen: end of utc late backport & config window
  • 21:30 brennen@deploy1002: Finished scap: Backport for Enable client preferences everywhere (T327979) (duration: 11m 14s)
  • 21:23 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage
  • 21:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS bullseye
  • 21:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
  • 21:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1077.eqiad.wmnet with OS bullseye
  • 21:21 brennen@deploy1002: brennen and nray: Backport for Enable client preferences everywhere (T327979) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet
  • 21:20 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage
  • 21:19 brennen@deploy1002: Started scap: Backport for Enable client preferences everywhere (T327979)
  • 21:18 brennen@deploy1002: Finished scap: Backport for Disable write old for CheckUserLog reason everywhere (T233004) (duration: 12m 02s)
  • 21:07 brennen@deploy1002: brennen and dreamyjazz: Backport for Disable write old for CheckUserLog reason everywhere (T233004) synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
  • 21:06 brennen@deploy1002: Started scap: Backport for Disable write old for CheckUserLog reason everywhere (T233004)
  • 20:59 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye
  • 20:59 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1078.eqiad.wmnet with OS bullseye
  • 20:52 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage
  • 20:49 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage
  • 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye
  • 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1077.eqiad.wmnet with OS bullseye
  • 20:23 rzl: rzl@apt1001:~$ sud