You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Server Admin Log: Difference between revisions
Jump to navigation
Jump to search
imported>Stashbot (mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .) |
imported>Stashbot (pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2435']) |
||
(425 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
== | == 2023-02-08 == | ||
* | * 01:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2435'] | ||
* 01:07 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2434'] | |||
* 01:00 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2435'] | |||
* 01:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2433'] | |||
* 01:00 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2434'] | |||
* 00:58 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2432'] | |||
* | * 00:52 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2433'] | ||
* 00:52 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2432'] | |||
* 00:50 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2431'] | |||
* | * 00:49 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2430'] | ||
* 00:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2431'] | |||
* 00:43 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2430'] | |||
* 00:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2429'] | |||
* | * 00:39 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2428'] | ||
* 00:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2429'] | |||
* 00:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2427'] | |||
* 00:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2428'] | |||
* 00:27 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['mw2426'] | |||
* | * 00:22 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2427'] | ||
* 00:17 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2426'] | |||
* 00:07 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mw2424'] | |||
* | * 00:06 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mw2425'] | ||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
* | |||
== | == 2023-02-07 == | ||
* | * 23:56 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2425'] | ||
* | * 23:56 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2424'] | ||
* 23:51 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2423'] | |||
* | * 23:49 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2422'] | ||
* 23:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2423'] | |||
* | * 23:32 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2422'] | ||
* | * 23:31 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2421'] | ||
* | * 23:30 pt1979@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['mw2420'] | ||
* | * 23:23 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2421'] | ||
* | * 23:22 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2420'] | ||
* | * 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2434.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2435.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:59 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2435.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 22:59 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2434.mgmt.codfw.wmnet with reboot policy FORCED | |||
* | * 22:56 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2432.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:56 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2433.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2433.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 22:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2432.mgmt.codfw.wmnet with reboot policy FORCED | |||
* | * 22:44 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* | * 22:44 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B8 - pt1979@cumin2002" | ||
* | * 22:43 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B8 - pt1979@cumin2002" | ||
* | * 22:41 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2430.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 22:41 pt1979@cumin2002: START - Cookbook sre.dns.netbox | |||
* | * 22:41 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2431.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:31 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2431.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:31 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2430.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:30 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2429.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2428.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2429.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 22:16 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2428.mgmt.codfw.wmnet with reboot policy FORCED | |||
* | * 22:15 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* | * 22:15 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B6 - pt1979@cumin2002" | ||
* 16: | * 22:14 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in B6 - pt1979@cumin2002" | ||
* 16: | * 22:12 pt1979@cumin2002: START - Cookbook sre.dns.netbox | ||
* | * 22:10 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "provision new Ganeti VM an-airflow1005 - bking@cumin1001 - [[phab:T327970|T327970]]" | ||
* | * 22:08 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:884333{{!}}Allow AbuseFilter to block IPs and users on itwikiversity (T328194)]] (duration: 08m 23s) | ||
* 22:07 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "provision new Ganeti VM an-airflow1005 - bking@cumin1001 - [[phab:T327970|T327970]]" | |||
* | * 22:02 urbanecm@deploy1002: urbanecm and superpes: Backport for [[gerrit:884333{{!}}Allow AbuseFilter to block IPs and users on itwikiversity (T328194)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet | ||
* 22:00 urbanecm@deploy1002: Started scap: Backport for [[gerrit:884333{{!}}Allow AbuseFilter to block IPs and users on itwikiversity (T328194)]] | |||
* | * 21:59 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886983{{!}}Change the trwiki logo with a temporary one (old vector) (T329047)]] (duration: 10m 20s) | ||
* 21:51 urbanecm@deploy1002: superpes and urbanecm: Backport for [[gerrit:886983{{!}}Change the trwiki logo with a temporary one (old vector) (T329047)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet | |||
* 21:49 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886983{{!}}Change the trwiki logo with a temporary one (old vector) (T329047)]] | |||
* | * 21:48 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886416{{!}}Install WikiLove extension on bnwikiquote (T328834)]] (duration: 15m 32s) | ||
* | * 21:35 urbanecm@deploy1002: superpes and urbanecm: Backport for [[gerrit:886416{{!}}Install WikiLove extension on bnwikiquote (T328834)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet | ||
* 21:34 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2051.codfw.wmnet with OS bullseye | |||
* 21:33 urbanecm: Create extension tables for Wikilove on bnwikiquote ([[phab:T328834|T328834]]) | |||
* | * 21:33 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886416{{!}}Install WikiLove extension on bnwikiquote (T328834)]] | ||
* 21:32 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED | |||
* | * 21:31 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:887353{{!}}Disable languages on history page (T328996)]], [[gerrit:887351{{!}}Remove button styling from log in link (T289212)]], [[gerrit:887350{{!}}[followup] mediawiki.feedlink: Atom's link icon overlaps the link (T327717)]] (duration: 11m 10s) | ||
* | * 21:29 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1053.eqiad.wmnet with OS bullseye | ||
* | * 21:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 21:24 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 21:22 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 21:22 urbanecm@deploy1002: urbanecm and jdlrobson: Backport for [[gerrit:887353{{!}}Disable languages on history page (T328996)]], [[gerrit:887351{{!}}Remove button styling from log in link (T289212)]], [[gerrit:887350{{!}}[followup] mediawiki.feedlink: Atom's link icon overlaps the link (T327717)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | ||
* 21:21 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED | |||
* | * 21:21 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 21:20 urbanecm@deploy1002: Started scap: Backport for [[gerrit:887353{{!}}Disable languages on history page (T328996)]], [[gerrit:887351{{!}}Remove button styling from log in link (T289212)]], [[gerrit:887350{{!}}[followup] mediawiki.feedlink: Atom's link icon overlaps the link (T327717)]] | ||
* | * 21:18 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2051.codfw.wmnet with reason: host reimage | ||
* | * 21:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2427.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 21:15 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1053.eqiad.wmnet with reason: host reimage | |||
* 21:14 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2051.codfw.wmnet with reason: host reimage | |||
* | * 21:12 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1053.eqiad.wmnet with reason: host reimage | ||
* | * 21:12 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2426.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 21:02 otto@deploy1002: Synchronized wmf-config/InitialiseSettings.php: wgEventSreams - Fix android session schema path (duration: 07m 26s) | ||
* | * 21:01 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1053.eqiad.wmnet with OS bullseye | ||
* 20:58 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2051.codfw.wmnet with OS bullseye | |||
* | * 20:57 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2050.codfw.wmnet with OS bullseye | ||
* | * 20:50 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1051.eqiad.wmnet with OS bullseye | ||
* | * 20:44 bking@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host an-airflow1005.eqiad.wmnet | ||
* | * 20:41 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2050.codfw.wmnet with reason: host reimage | ||
* | * 20:38 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2050.codfw.wmnet with reason: host reimage | ||
* | * 20:36 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1051.eqiad.wmnet with reason: host reimage | ||
* | * 20:33 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1051.eqiad.wmnet with reason: host reimage | ||
* 20:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2050.codfw.wmnet with OS bullseye | |||
* | * 20:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1051.eqiad.wmnet with OS bullseye | ||
* 20:13 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED | |||
* | * 20:09 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 20:08 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 20:04 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:59 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:58 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 19:57 bking@cumin1001: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-airflow1005.eqiad.wmnet on all recursors | |||
* | * 19:57 bking@cumin1001: START - Cookbook sre.dns.wipe-cache an-airflow1005.eqiad.wmnet on all recursors | ||
* 19:57 bking@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* | * 19:57 bking@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM an-airflow1005.eqiad.wmnet - bking@cumin1001" | ||
* | * 19:56 bking@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM an-airflow1005.eqiad.wmnet - bking@cumin1001" | ||
* | * 19:55 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:55 demon@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.22 refs [[phab:T325585|T325585]] | ||
* | * 19:54 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 19:53 bking@cumin1001: START - Cookbook sre.dns.netbox | |||
* | * 19:53 bking@cumin1001: START - Cookbook sre.ganeti.makevm for new host an-airflow1005.eqiad.wmnet | ||
* | * 19:48 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2425.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:47 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:47 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:46 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:45 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 19:39 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2049.codfw.wmnet with OS bullseye | ||
* | * 19:33 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1049.eqiad.wmnet with OS bullseye | ||
* | * 19:23 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2049.codfw.wmnet with reason: host reimage | ||
* | * 19:20 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2049.codfw.wmnet with reason: host reimage | ||
* 19:18 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1049.eqiad.wmnet with reason: host reimage | |||
* 19:15 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1049.eqiad.wmnet with reason: host reimage | |||
* 19:04 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1049.eqiad.wmnet with OS bullseye | |||
* 19:03 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2049.codfw.wmnet with OS bullseye | |||
* 19:03 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2423.mgmt.codfw.wmnet with reboot policy FORCED | |||
* 19:01 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2422.mgmt.codfw.wmnet with reboot policy FORCED | |||
* 19:00 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 19:00 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2423,25,26,27 DNS - pt1979@cumin2002" | |||
* 19:00 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2423,25,26,27 DNS - pt1979@cumin2002" | |||
* 18:57 pt1979@cumin2002: START - Cookbook sre.dns.netbox | |||
* 18:53 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2048.codfw.wmnet with OS bullseye | |||
* 18:47 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1047.eqiad.wmnet with OS bullseye | |||
* 18:37 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2048.codfw.wmnet with reason: host reimage | |||
* 18:34 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2048.codfw.wmnet with reason: host reimage | |||
* 18:32 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1047.eqiad.wmnet with reason: host reimage | |||
* 18:29 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1047.eqiad.wmnet with reason: host reimage | |||
* 18:18 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2048.codfw.wmnet with OS bullseye | |||
* 18:17 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1047.eqiad.wmnet with OS bullseye | |||
* 18:02 bking@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts | |||
* 18:02 bking@cumin2002: START - Cookbook sre.hosts.remove-downtime for 13 hosts | |||
* 17:55 inflatador: bking@cumin1001 repooling elastic and wdqs hosts post-maintenance [[phab:T327925|T327925]] | |||
* 17:53 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2047.codfw.wmnet with OS bullseye | |||
* 17:51 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1046.eqiad.wmnet with OS bullseye | |||
* 17:40 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2047.codfw.wmnet with reason: host reimage | |||
* 17:37 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2047.codfw.wmnet with reason: host reimage | |||
* 17:37 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1046.eqiad.wmnet with reason: host reimage | |||
* 17:34 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1046.eqiad.wmnet with reason: host reimage | |||
* 17:22 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1046.eqiad.wmnet with OS bullseye | |||
* 17:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2047.codfw.wmnet with OS bullseye | |||
* 16:50 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2046.codfw.wmnet with OS bullseye | |||
* 16:48 urbanecm@deploy1002: Finished scap: {{Gerrit|58f4d877}}: Finalize mediawiki/page/change schema, produce at rc1.mediawiki.page_change ([[phab:T308017|T308017]]), {{Gerrit|854ff4ac}}: Finalize mediawiki/page/change schema at 1.0.0 ([[phab:T308017|T308017]]) (duration: 07m 32s) | |||
* 16:46 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1045.eqiad.wmnet with OS bullseye | |||
* 16:41 urbanecm@deploy1002: Started scap: {{Gerrit|58f4d877}}: Finalize mediawiki/page/change schema, produce at rc1.mediawiki.page_change ([[phab:T308017|T308017]]), {{Gerrit|854ff4ac}}: Finalize mediawiki/page/change schema at 1.0.0 ([[phab:T308017|T308017]]) | |||
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43765 and previous config saved to /var/cache/conftool/dbconfig/20230207-163902-root.json | |||
* 16:34 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage | |||
* 16:31 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage | |||
* 16:31 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1045.eqiad.wmnet with reason: host reimage | |||
* 16:26 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1045.eqiad.wmnet with reason: host reimage | |||
* 16:23 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43764 and previous config saved to /var/cache/conftool/dbconfig/20230207-162357-root.json | |||
* 16:18 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886985{{!}}Restore mediawiki.page-undelete hook (T329064)]], [[gerrit:887346{{!}}Restore mediawiki.page-undelete hook (T329064)]] (duration: 17m 44s) | |||
* 16:15 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2046.codfw.wmnet with OS bullseye | |||
* 16:14 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1045.eqiad.wmnet with OS bullseye | |||
* 16:08 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43763 and previous config saved to /var/cache/conftool/dbconfig/20230207-160852-root.json | |||
* 16:02 urbanecm@deploy1002: urbanecm: Backport for [[gerrit:886985{{!}}Restore mediawiki.page-undelete hook (T329064)]], [[gerrit:887346{{!}}Restore mediawiki.page-undelete hook (T329064)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* 16:00 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886985{{!}}Restore mediawiki.page-undelete hook (T329064)]], [[gerrit:887346{{!}}Restore mediawiki.page-undelete hook (T329064)]] | |||
* 15:53 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43762 and previous config saved to /var/cache/conftool/dbconfig/20230207-155347-root.json | |||
* 15:53 moritzm: installing tiff security updates | |||
* 15:48 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2045.codfw.wmnet with OS bullseye | |||
* 15:47 urbanecm@deploy1002: Finished scap: {{Gerrit|20a79c55b7073e791e297a5389fa66819f596178}}: Don't add custom attributes in unwrapParsoidSections() ([[phab:T328268|T328268]]) (duration: 07m 34s) | |||
* 15:43 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1043.eqiad.wmnet with OS bullseye | |||
* 15:39 urbanecm@deploy1002: Started scap: {{Gerrit|20a79c55b7073e791e297a5389fa66819f596178}}: Don't add custom attributes in unwrapParsoidSections() ([[phab:T328268|T328268]]) | |||
* 15:38 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43761 and previous config saved to /var/cache/conftool/dbconfig/20230207-153842-root.json | |||
* 15:32 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2045.codfw.wmnet with reason: host reimage | |||
* 15:29 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2045.codfw.wmnet with reason: host reimage | |||
* 15:28 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1043.eqiad.wmnet with reason: host reimage | |||
* 15:26 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886997{{!}}Add "Page Frame" to DiscussionTools beta feature on enwiki (T327456)]] (duration: 10m 39s) | |||
* 15:25 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1043.eqiad.wmnet with reason: host reimage | |||
* 15:23 marostegui@cumin1001: dbctl commit (dc=all): 'db1187 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43760 and previous config saved to /var/cache/conftool/dbconfig/20230207-152337-root.json | |||
* 15:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1003.eqiad.wmnet | |||
* 15:17 urbanecm@deploy1002: matmarex and urbanecm: Backport for [[gerrit:886997{{!}}Add "Page Frame" to DiscussionTools beta feature on enwiki (T327456)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet | |||
* 15:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host people1003.eqiad.wmnet | |||
* 15:15 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886997{{!}}Add "Page Frame" to DiscussionTools beta feature on enwiki (T327456)]] | |||
* 15:14 volans@cumin2002: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool restbase-async in eqiad: [[phab:T327925|T327925]] | |||
* 15:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet | |||
* 15:13 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1043.eqiad.wmnet with OS bullseye | |||
* 15:13 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2045.codfw.wmnet with OS bullseye | |||
* 15:12 vgutierrez: repool codfw edge site - [[phab:T327925|T327925]] | |||
* 15:09 volans@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) restbase-async.discovery.wmnet on all recursors | |||
* 15:09 volans@cumin2002: START - Cookbook sre.dns.wipe-cache restbase-async.discovery.wmnet on all recursors | |||
* 15:09 volans@cumin2002: START - Cookbook sre.discovery.service-route depool restbase-async in eqiad: [[phab:T327925|T327925]] | |||
* 15:08 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet | |||
* 15:07 volans@cumin2002: END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) pool all active/active services in codfw: [[phab:T327925|T327925]] | |||
* 15:05 marostegui: dbmaint deploy schema change on s8 [[phab:T328807|T328807]] [[phab:T328828|T328828]] | |||
* 15:04 vgutierrez: restart pybal in lvs2010 - [[phab:T327925|T327925]] | |||
* 15:01 marostegui: dbmaint deploy schema change on s6 [[phab:T328807|T328807]] | |||
* 15:00 vgutierrez: restart pybal in lvs2009 - [[phab:T327925|T327925]] | |||
* 14:59 marostegui: dbmaint deploy schema change on s6 [[phab:T328828|T328828]] | |||
* 14:53 moritzm: adding nfraison to pwstore [[phab:T328915|T328915]] | |||
* 14:46 volans@cumin2002: START - Cookbook sre.discovery.datacenter-route pool all active/active services in codfw: [[phab:T327925|T327925]] | |||
* 14:40 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=thanos-fe2002.codfw.wmnet,service=thanos-web | |||
* 14:40 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=thanos-fe2001.codfw.wmnet,service=thanos-web | |||
* 14:36 claime: repooled appserver, api_appserver, jobrunner, parsoid - [[phab:T327925|T327925]] | |||
* 14:36 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) | |||
* 14:36 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=api_appserver | |||
* 14:35 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=jobrunner | |||
* 14:35 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=appserver | |||
* 14:35 cgoubert@cumin1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid | |||
* 14:32 Emperor: pool ms-fe2009 (codfw as a whole still depooled) [[phab:T327925|T327925]] | |||
* 14:28 jbond: enable puppet in codfw, uslfo, esams post switch upgrade [[phab:T327925|T327925]] | |||
* 14:26 claime: depooled appserver, api_appserver, jobrunner, parsoid - [[phab:T327925|T327925]] | |||
* 14:25 mvernon@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) | |||
* 14:21 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=parsoid | |||
* 14:19 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=appserver | |||
* 14:19 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=jobrunner | |||
* 14:18 cgoubert@cumin1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=api_appserver | |||
* 14:13 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=thanos-fe2002.codfw.wmnet,service=thanos-web | |||
* 14:13 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=thanos-fe2001.codfw.wmnet,service=thanos-web | |||
* 14:08 jbond: disable puppet in codfw, uslfo, esams for switch upgrade [[phab:T327925|T327925]] | |||
* 14:07 lucaswerkmeister-wmde@deploy1002: backport aborted: (duration: 17m 46s) | |||
* 14:06 XioNoX: asw-a-codfw> request system reboot all-members - [[phab:T327925|T327925]] | |||
* 13:59 XioNoX: disable puppet in ulsfo/esams/codfw for codfw row A switch upgrade - [[phab:T327925|T327925]] | |||
* 13:56 Emperor: depool ms-fe2009 [[phab:T327925|T327925]] | |||
* 13:55 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 13:55 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2422 and 24 DNS - pt1979@cumin2002" | |||
* 13:54 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2422 and 24 DNS - pt1979@cumin2002" | |||
* 13:51 pt1979@cumin2002: START - Cookbook sre.dns.netbox | |||
* 13:33 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 199 hosts with reason: codfw row A upgrade | |||
* 13:32 oblivian@cumin2002: END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) depool all active/active services in codfw: [[phab:T327925|T327925]] | |||
* 13:31 vgutierrez: depool codfw edge site - [[phab:T327925|T327925]] | |||
* 13:31 ayounsi@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on 199 hosts with reason: codfw row A upgrade | |||
* 13:13 jbond: enable puppet in codfw, ulsfo and esams to allow depools post switch upgrade [[phab:T327925|T327925]] | |||
* 13:11 oblivian@cumin2002: START - Cookbook sre.discovery.datacenter-route depool all active/active services in codfw: [[phab:T327925|T327925]] | |||
* 13:05 jbond: diable puppet in codfw, ulsfo and esams for switch upgrade [[phab:T327925|T327925]] | |||
* 12:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm6001.drmrs.wmnet | |||
* 12:28 vgutierrez: depooling authdns2001 - [[phab:T327925|T327925]] | |||
* 12:25 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on doh2001.wikimedia.org with reason: depooled; [[phab:T327925|T327925]] | |||
* 12:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 8:00:00 on doh2001.wikimedia.org with reason: depooled; [[phab:T327925|T327925]] | |||
* 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm6001.drmrs.wmnet on all recursors | |||
* 12:20 jmm@cumin2002: START - Cookbook sre.dns.wipe-cache testvm6001.drmrs.wmnet on all recursors | |||
* 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 12:20 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm6001.drmrs.wmnet - jmm@cumin2002" | |||
* 12:19 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm6001.drmrs.wmnet - jmm@cumin2002" | |||
* 12:17 jmm@cumin2002: START - Cookbook sre.dns.netbox | |||
* 12:17 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host testvm6001.drmrs.wmnet | |||
* 12:00 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1041.eqiad.wmnet with OS bullseye | |||
* 11:56 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2044.codfw.wmnet with OS bullseye | |||
* 11:56 marostegui: Install 10.4.28 on db1152 [[phab:T329011|T329011]] | |||
* 11:52 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons. | |||
* 11:44 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1041.eqiad.wmnet with reason: host reimage | |||
* 11:41 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1041.eqiad.wmnet with reason: host reimage | |||
* 11:40 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2044.codfw.wmnet with reason: host reimage | |||
* 11:37 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2044.codfw.wmnet with reason: host reimage | |||
* 11:33 moritzm: installing imagemagick security updates on buster | |||
* 11:29 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1041.eqiad.wmnet with OS bullseye | |||
* 11:21 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2044.codfw.wmnet with OS bullseye | |||
* 10:51 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-logging-eqiad cluster: Roll restart of jvm daemons. | |||
* 10:49 elukey@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-logging-codfw cluster: Roll restart of jvm daemons. | |||
* 10:19 oblivian@cumin2002: END (PASS) - Cookbook sre.discovery.datacenter-route (exit_code=0) pool all active/active services in eqiad: Pooling eqiad for codfw depool today | |||
* 10:19 oblivian@cumin2002: START - Cookbook sre.discovery.datacenter-route pool all active/active services in eqiad: Pooling eqiad for codfw depool today | |||
* 10:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast1003.wikimedia.org with OS bullseye | |||
* 10:13 oblivian@cumin2002: END (FAIL) - Cookbook sre.discovery.datacenter-route (exit_code=93) pool all active/active services in eqiad: Pooling eqiad for codfw depool today | |||
* 10:12 oblivian@cumin2002: START - Cookbook sre.discovery.datacenter-route pool all active/active services in eqiad: Pooling eqiad for codfw depool today | |||
* 10:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast1003.wikimedia.org with reason: host reimage | |||
* 09:56 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast1003.wikimedia.org with reason: host reimage | |||
* 09:44 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast1003.wikimedia.org with OS bullseye | |||
* 09:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host bast2002.wikimedia.org with OS bullseye | |||
* 09:24 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync | |||
* 09:23 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: sync | |||
* 09:22 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast2002.wikimedia.org with reason: host reimage | |||
* 09:20 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: sync | |||
* 09:20 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: sync | |||
* 09:20 akosiaris: add wiktionary to mobile-sections rerenders. [[phab:T226931|T226931]] | |||
* 09:19 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on bast2002.wikimedia.org with reason: host reimage | |||
* 09:19 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: sync | |||
* 09:19 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: sync | |||
* 09:08 elukey@cumin1001: START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-logging-codfw cluster: Roll restart of jvm daemons. | |||
* 09:02 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host bast2002.wikimedia.org with OS bullseye | |||
* 08:50 vgutierrez: rolling upgrade to HAProxy 2.4.21 in cp nodes | |||
* 08:48 kostajh: UTC morning deploys done | |||
* 08:48 kharlan@deploy1002: Finished scap: Backport for [[gerrit:883236{{!}}[Growth] Remove mentor list variables (T321501)]], [[gerrit:883153{{!}}Remove GEMentorProvider (T321501)]] (duration: 12m 48s) | |||
* 08:37 kharlan@deploy1002: urbanecm and kharlan: Backport for [[gerrit:883236{{!}}[Growth] Remove mentor list variables (T321501)]], [[gerrit:883153{{!}}Remove GEMentorProvider (T321501)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet | |||
* 08:35 kharlan@deploy1002: Started scap: Backport for [[gerrit:883236{{!}}[Growth] Remove mentor list variables (T321501)]], [[gerrit:883153{{!}}Remove GEMentorProvider (T321501)]] | |||
* 08:30 moritzm: installing imagemagick security updates on Thumbor [[phab:T328901|T328901]] | |||
* 08:28 kharlan@deploy1002: Finished scap: Backport for [[gerrit:886343{{!}}GrowthExperiments: Disable leveling up features in production (T328757)]] (duration: 12m 11s) | |||
* 08:18 kharlan@deploy1002: kharlan: Backport for [[gerrit:886343{{!}}GrowthExperiments: Disable leveling up features in production (T328757)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet | |||
* 08:16 kharlan@deploy1002: Started scap: Backport for [[gerrit:886343{{!}}GrowthExperiments: Disable leveling up features in production (T328757)]] | |||
* 08:14 kharlan@deploy1002: backport aborted: (duration: 00m 07s) | |||
* 07:00 marostegui: Failover m3 from db1159 to db1164 - [[phab:T328404|T328404]] | |||
* 06:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2110 in API', diff saved to https://phabricator.wikimedia.org/P43758 and previous config saved to /var/cache/conftool/dbconfig/20230207-063147-root.json | |||
* 06:28 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1187', diff saved to https://phabricator.wikimedia.org/P43757 and previous config saved to /var/cache/conftool/dbconfig/20230207-062826-root.json | |||
* 04:58 mwpresync@deploy1002: Pruned MediaWiki: 1.40.0-wmf.20 (duration: 02m 20s) | |||
* 04:55 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.40.0-wmf.22 refs [[phab:T325585|T325585]] (duration: 53m 11s) | |||
* 04:02 mwpresync@deploy1002: Started scap: testwikis wikis to 1.40.0-wmf.22 refs [[phab:T325585|T325585]] | |||
== | == 2023-02-06 == | ||
* 23: | * 23:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 23: | * 23:01 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:55 ryankemper: [[phab:T327925|T327925]] Depooled codfw wdqs hosts: `ryankemper@cumin2002:~$ sudo -E cumin -b 3 'wdqs[2003-2004,2009]*' 'sudo depool'` | ||
* 22: | * 22:51 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 13 hosts with reason: switch upgrade | ||
* 22:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 13 hosts with reason: switch upgrade | |||
* 22:48 ryankemper: [[phab:T327925|T327925]] Banned `elastic[2037-2040,2055-2056,2061-2062,2069,2073-2076]` on codfw elastic | |||
* 22: | * 22:42 inflatador: bking@cumin2002 banning Elastic nodes from cluster in preparation for [[phab:T327925|T327925]] | ||
* 22: | * 22:17 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:10 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2421.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:08 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mw2421 | ||
* | * 22:07 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mw2421 | ||
* | * 22:06 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* 22:06 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2421 DNS - pt1979@cumin2002" | |||
* | * 22:05 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2421 DNS - pt1979@cumin2002" | ||
* | * 22:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED | ||
* | * 22:01 pt1979@cumin2002: START - Cookbook sre.dns.netbox | ||
* | * 22:00 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 19: | * 19:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 19:32 zabe@deploy1002: say aborted: (duration: 00m 39s) | |||
* 19: | * 19:30 zabe@deploy1002: backport aborted: (duration: 00m 00s) | ||
* 19: | * 19:29 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 92.62.231.190 # [[phab:T328929|T328929]] | ||
* 19: | * 19:27 zabe@deploy1002: backport aborted: (duration: 00m 23s) | ||
* 19: | * 19:25 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886910{{!}}Add a new throttle rule (T328929)]] (duration: 07m 43s) | ||
* 19: | * 19:18 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886910{{!}}Add a new throttle rule (T328929)]] | ||
* 18: | * 19:17 urbanecm@deploy1002: backport aborted: (duration: 00m 01s) | ||
* 18: | * 18:53 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host mw2420.mgmt.codfw.wmnet with reboot policy FORCED | ||
* 18: | * 18:52 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* 18:52 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2420 DNS - pt1979@cumin2002" | |||
* 18: | * 18:51 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mw2420 DNS - pt1979@cumin2002" | ||
* 18: | * 18:51 pt1979@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mw2420 | ||
* 18: | * 18:50 pt1979@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mw2420 | ||
* 18: | * 18:48 pt1979@cumin2002: START - Cookbook sre.dns.netbox | ||
* 18: | * 18:48 pt1979@cumin2002: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) | ||
* | * 18:48 pt1979@cumin2002: START - Cookbook sre.dns.netbox | ||
* 15:10 vgutierrez: rolling upgrade to HAProxy 2.4.21 in ulsfo cp nodes | |||
* | * 14:37 moritzm: installing imagemagick security updates on buster | ||
* | * 14:13 vgutierrez: testing HAProxy 2.4.21 in cp4052 and cp4044 | ||
* | * 14:11 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:881918{{!}}New config entries for migrated android schemas (T324167)]] (duration: 09m 19s) | ||
* 14:09 vgutierrez: fetch HAProxy 2.4.21 for buster and bullseye (apt.wm.o) | |||
* | * 14:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43754 and previous config saved to /var/cache/conftool/dbconfig/20230206-140753-root.json | ||
* | * 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43753 and previous config saved to /var/cache/conftool/dbconfig/20230206-140627-root.json | ||
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43752 and previous config saved to /var/cache/conftool/dbconfig/20230206-140623-root.json | |||
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43751 and previous config saved to /var/cache/conftool/dbconfig/20230206-140606-root.json | |||
* 14:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43750 and previous config saved to /var/cache/conftool/dbconfig/20230206-140602-root.json | |||
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43749 and previous config saved to /var/cache/conftool/dbconfig/20230206-140554-root.json | |||
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43748 and previous config saved to /var/cache/conftool/dbconfig/20230206-140549-root.json | |||
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43747 and previous config saved to /var/cache/conftool/dbconfig/20230206-140541-root.json | |||
* | * 14:05 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided) (duration: 00m 33s) | ||
* 14:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43746 and previous config saved to /var/cache/conftool/dbconfig/20230206-140501-root.json | |||
* 14:05 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided) | |||
* | * 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43745 and previous config saved to /var/cache/conftool/dbconfig/20230206-140449-root.json | ||
* | * 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43744 and previous config saved to /var/cache/conftool/dbconfig/20230206-140433-root.json | ||
* 14:04 urbanecm@deploy1002: urbanecm and sharvaniharan: Backport for [[gerrit:881918{{!}}New config entries for migrated android schemas (T324167)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* 14:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43743 and previous config saved to /var/cache/conftool/dbconfig/20230206-140405-root.json | |||
* | * 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43742 and previous config saved to /var/cache/conftool/dbconfig/20230206-140338-root.json | ||
* | * 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43741 and previous config saved to /var/cache/conftool/dbconfig/20230206-140333-root.json | ||
* | * 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43740 and previous config saved to /var/cache/conftool/dbconfig/20230206-140316-root.json | ||
* 14:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43739 and previous config saved to /var/cache/conftool/dbconfig/20230206-140310-root.json | |||
* | * 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43738 and previous config saved to /var/cache/conftool/dbconfig/20230206-140257-root.json | ||
* 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43737 and previous config saved to /var/cache/conftool/dbconfig/20230206-140249-root.json | |||
* | * 14:02 urbanecm@deploy1002: Started scap: Backport for [[gerrit:881918{{!}}New config entries for migrated android schemas (T324167)]] | ||
* | * 13:57 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 3300 | ||
* 13:56 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 3300 | |||
* 14: | * 13:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43736 and previous config saved to /var/cache/conftool/dbconfig/20230206-135248-root.json | ||
* 14: | * 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43735 and previous config saved to /var/cache/conftool/dbconfig/20230206-135122-root.json | ||
* 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43734 and previous config saved to /var/cache/conftool/dbconfig/20230206-135118-root.json | |||
* 14: | * 13:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43733 and previous config saved to /var/cache/conftool/dbconfig/20230206-135101-root.json | ||
* 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43732 and previous config saved to /var/cache/conftool/dbconfig/20230206-135057-root.json | |||
* 14: | * 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43731 and previous config saved to /var/cache/conftool/dbconfig/20230206-135049-root.json | ||
* 14: | * 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43730 and previous config saved to /var/cache/conftool/dbconfig/20230206-135044-root.json | ||
* 13:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43729 and previous config saved to /var/cache/conftool/dbconfig/20230206-135036-root.json | |||
* | * 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43728 and previous config saved to /var/cache/conftool/dbconfig/20230206-134956-root.json | ||
* | * 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43727 and previous config saved to /var/cache/conftool/dbconfig/20230206-134944-root.json | ||
* | * 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43726 and previous config saved to /var/cache/conftool/dbconfig/20230206-134928-root.json | ||
* | * 13:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43725 and previous config saved to /var/cache/conftool/dbconfig/20230206-134901-root.json | ||
* 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43724 and previous config saved to /var/cache/conftool/dbconfig/20230206-134833-root.json | |||
* | * 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43723 and previous config saved to /var/cache/conftool/dbconfig/20230206-134828-root.json | ||
* | * 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43722 and previous config saved to /var/cache/conftool/dbconfig/20230206-134811-root.json | ||
* 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43721 and previous config saved to /var/cache/conftool/dbconfig/20230206-134805-root.json | |||
* 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43720 and previous config saved to /var/cache/conftool/dbconfig/20230206-134752-root.json | |||
* 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43719 and previous config saved to /var/cache/conftool/dbconfig/20230206-134744-root.json | |||
* 13:37 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43718 and previous config saved to /var/cache/conftool/dbconfig/20230206-133743-root.json | |||
* 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43717 and previous config saved to /var/cache/conftool/dbconfig/20230206-133618-root.json | |||
* 13:36 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43716 and previous config saved to /var/cache/conftool/dbconfig/20230206-133613-root.json | |||
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43715 and previous config saved to /var/cache/conftool/dbconfig/20230206-133556-root.json | |||
* | * 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43714 and previous config saved to /var/cache/conftool/dbconfig/20230206-133552-root.json | ||
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43713 and previous config saved to /var/cache/conftool/dbconfig/20230206-133544-root.json | |||
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43712 and previous config saved to /var/cache/conftool/dbconfig/20230206-133540-root.json | |||
* | * 13:35 jbond: add confd to bookworm repos | ||
* 13:35 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43711 and previous config saved to /var/cache/conftool/dbconfig/20230206-133531-root.json | |||
* 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43710 and previous config saved to /var/cache/conftool/dbconfig/20230206-133451-root.json | |||
* 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43709 and previous config saved to /var/cache/conftool/dbconfig/20230206-133439-root.json | |||
* | * 13:34 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43708 and previous config saved to /var/cache/conftool/dbconfig/20230206-133423-root.json | ||
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43707 and previous config saved to /var/cache/conftool/dbconfig/20230206-133356-root.json | |||
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43706 and previous config saved to /var/cache/conftool/dbconfig/20230206-133329-root.json | |||
* | * 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43705 and previous config saved to /var/cache/conftool/dbconfig/20230206-133323-root.json | ||
* 04 | * 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43704 and previous config saved to /var/cache/conftool/dbconfig/20230206-133306-root.json | ||
* 13:33 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43703 and previous config saved to /var/cache/conftool/dbconfig/20230206-133300-root.json | |||
* 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43702 and previous config saved to /var/cache/conftool/dbconfig/20230206-133247-root.json | |||
* 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43701 and previous config saved to /var/cache/conftool/dbconfig/20230206-133239-root.json | |||
* 13:26 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | |||
* 13:26 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | |||
* 13:23 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | |||
* 13:22 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43700 and previous config saved to /var/cache/conftool/dbconfig/20230206-132238-root.json | |||
* 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43699 and previous config saved to /var/cache/conftool/dbconfig/20230206-132113-root.json | |||
* 13:21 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43698 and previous config saved to /var/cache/conftool/dbconfig/20230206-132108-root.json | |||
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43697 and previous config saved to /var/cache/conftool/dbconfig/20230206-132051-root.json | |||
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43696 and previous config saved to /var/cache/conftool/dbconfig/20230206-132047-root.json | |||
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43695 and previous config saved to /var/cache/conftool/dbconfig/20230206-132039-root.json | |||
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43694 and previous config saved to /var/cache/conftool/dbconfig/20230206-132035-root.json | |||
* 13:20 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43693 and previous config saved to /var/cache/conftool/dbconfig/20230206-132026-root.json | |||
* 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43692 and previous config saved to /var/cache/conftool/dbconfig/20230206-131947-root.json | |||
* 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43691 and previous config saved to /var/cache/conftool/dbconfig/20230206-131934-root.json | |||
* 13:19 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43690 and previous config saved to /var/cache/conftool/dbconfig/20230206-131918-root.json | |||
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43689 and previous config saved to /var/cache/conftool/dbconfig/20230206-131851-root.json | |||
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43688 and previous config saved to /var/cache/conftool/dbconfig/20230206-131824-root.json | |||
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43687 and previous config saved to /var/cache/conftool/dbconfig/20230206-131818-root.json | |||
* 13:18 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43686 and previous config saved to /var/cache/conftool/dbconfig/20230206-131801-root.json | |||
* 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43685 and previous config saved to /var/cache/conftool/dbconfig/20230206-131755-root.json | |||
* 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43684 and previous config saved to /var/cache/conftool/dbconfig/20230206-131740-root.json | |||
* 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43683 and previous config saved to /var/cache/conftool/dbconfig/20230206-131734-root.json | |||
* 13:07 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43682 and previous config saved to /var/cache/conftool/dbconfig/20230206-130733-root.json | |||
* 13:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43681 and previous config saved to /var/cache/conftool/dbconfig/20230206-130608-root.json | |||
* 13:06 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43680 and previous config saved to /var/cache/conftool/dbconfig/20230206-130603-root.json | |||
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43679 and previous config saved to /var/cache/conftool/dbconfig/20230206-130547-root.json | |||
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43678 and previous config saved to /var/cache/conftool/dbconfig/20230206-130542-root.json | |||
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43677 and previous config saved to /var/cache/conftool/dbconfig/20230206-130534-root.json | |||
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43676 and previous config saved to /var/cache/conftool/dbconfig/20230206-130530-root.json | |||
* 13:05 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43675 and previous config saved to /var/cache/conftool/dbconfig/20230206-130521-root.json | |||
* 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43674 and previous config saved to /var/cache/conftool/dbconfig/20230206-130442-root.json | |||
* 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43673 and previous config saved to /var/cache/conftool/dbconfig/20230206-130429-root.json | |||
* 13:04 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43672 and previous config saved to /var/cache/conftool/dbconfig/20230206-130414-root.json | |||
* 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43671 and previous config saved to /var/cache/conftool/dbconfig/20230206-130346-root.json | |||
* 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43670 and previous config saved to /var/cache/conftool/dbconfig/20230206-130319-root.json | |||
* 13:03 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43669 and previous config saved to /var/cache/conftool/dbconfig/20230206-130313-root.json | |||
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43668 and previous config saved to /var/cache/conftool/dbconfig/20230206-130256-root.json | |||
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43667 and previous config saved to /var/cache/conftool/dbconfig/20230206-130250-root.json | |||
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43666 and previous config saved to /var/cache/conftool/dbconfig/20230206-130235-root.json | |||
* 13:02 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43665 and previous config saved to /var/cache/conftool/dbconfig/20230206-130230-root.json | |||
* 12:52 marostegui@cumin1001: dbctl commit (dc=all): 'db2110 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43664 and previous config saved to /var/cache/conftool/dbconfig/20230206-125228-root.json | |||
* 12:51 marostegui@cumin1001: dbctl commit (dc=all): 'db2176 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43663 and previous config saved to /var/cache/conftool/dbconfig/20230206-125103-root.json | |||
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2175 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43662 and previous config saved to /var/cache/conftool/dbconfig/20230206-125059-root.json | |||
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43661 and previous config saved to /var/cache/conftool/dbconfig/20230206-125042-root.json | |||
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43660 and previous config saved to /var/cache/conftool/dbconfig/20230206-125037-root.json | |||
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43659 and previous config saved to /var/cache/conftool/dbconfig/20230206-125029-root.json | |||
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2155 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43658 and previous config saved to /var/cache/conftool/dbconfig/20230206-125025-root.json | |||
* 12:50 marostegui@cumin1001: dbctl commit (dc=all): 'db2154 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43657 and previous config saved to /var/cache/conftool/dbconfig/20230206-125017-root.json | |||
* 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2153 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43656 and previous config saved to /var/cache/conftool/dbconfig/20230206-124937-root.json | |||
* 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43655 and previous config saved to /var/cache/conftool/dbconfig/20230206-124924-root.json | |||
* 12:49 marostegui@cumin1001: dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43654 and previous config saved to /var/cache/conftool/dbconfig/20230206-124909-root.json | |||
* 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43653 and previous config saved to /var/cache/conftool/dbconfig/20230206-124841-root.json | |||
* 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2122 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43652 and previous config saved to /var/cache/conftool/dbconfig/20230206-124814-root.json | |||
* 12:48 marostegui@cumin1001: dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43651 and previous config saved to /var/cache/conftool/dbconfig/20230206-124808-root.json | |||
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43650 and previous config saved to /var/cache/conftool/dbconfig/20230206-124751-root.json | |||
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43649 and previous config saved to /var/cache/conftool/dbconfig/20230206-124745-root.json | |||
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2104 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43648 and previous config saved to /var/cache/conftool/dbconfig/20230206-124730-root.json | |||
* 12:47 marostegui@cumin1001: dbctl commit (dc=all): 'db2103 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43647 and previous config saved to /var/cache/conftool/dbconfig/20230206-124725-root.json | |||
* 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43646 and previous config saved to /var/cache/conftool/dbconfig/20230206-124629-root.json | |||
* 12:46 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43645 and previous config saved to /var/cache/conftool/dbconfig/20230206-124617-root.json | |||
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43644 and previous config saved to /var/cache/conftool/dbconfig/20230206-124513-root.json | |||
* 12:45 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43643 and previous config saved to /var/cache/conftool/dbconfig/20230206-124506-root.json | |||
* 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43642 and previous config saved to /var/cache/conftool/dbconfig/20230206-123124-root.json | |||
* 12:31 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43641 and previous config saved to /var/cache/conftool/dbconfig/20230206-123112-root.json | |||
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43640 and previous config saved to /var/cache/conftool/dbconfig/20230206-123007-root.json | |||
* 12:30 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43639 and previous config saved to /var/cache/conftool/dbconfig/20230206-123001-root.json | |||
* 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43638 and previous config saved to /var/cache/conftool/dbconfig/20230206-121619-root.json | |||
* 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43637 and previous config saved to /var/cache/conftool/dbconfig/20230206-121608-root.json | |||
* 12:15 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43636 and previous config saved to /var/cache/conftool/dbconfig/20230206-121503-root.json | |||
* 12:14 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43635 and previous config saved to /var/cache/conftool/dbconfig/20230206-121456-root.json | |||
* 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43634 and previous config saved to /var/cache/conftool/dbconfig/20230206-120114-root.json | |||
* 12:01 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43633 and previous config saved to /var/cache/conftool/dbconfig/20230206-120103-root.json | |||
* 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43631 and previous config saved to /var/cache/conftool/dbconfig/20230206-115958-root.json | |||
* 11:59 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43630 and previous config saved to /var/cache/conftool/dbconfig/20230206-115951-root.json | |||
* 11:58 btullis@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host db1108.eqiad.wmnet | |||
* 11:47 jbond: puppetmaster[12]002 reintroduced to services | |||
* 11:46 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host db1108.eqiad.wmnet | |||
* 11:46 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43629 and previous config saved to /var/cache/conftool/dbconfig/20230206-114609-root.json | |||
* 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43628 and previous config saved to /var/cache/conftool/dbconfig/20230206-114558-root.json | |||
* 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43627 and previous config saved to /var/cache/conftool/dbconfig/20230206-114453-root.json | |||
* 11:44 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43626 and previous config saved to /var/cache/conftool/dbconfig/20230206-114446-root.json | |||
* 11:31 marostegui@cumin1001: dbctl commit (dc=all): 'db2156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43625 and previous config saved to /var/cache/conftool/dbconfig/20230206-113104-root.json | |||
* 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'db2105 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43624 and previous config saved to /var/cache/conftool/dbconfig/20230206-113053-root.json | |||
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'db2126 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43623 and previous config saved to /var/cache/conftool/dbconfig/20230206-112948-root.json | |||
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'db2130 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43622 and previous config saved to /var/cache/conftool/dbconfig/20230206-112942-root.json | |||
* 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43621 and previous config saved to /var/cache/conftool/dbconfig/20230206-112900-root.json | |||
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43620 and previous config saved to /var/cache/conftool/dbconfig/20230206-112856-root.json | |||
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43619 and previous config saved to /var/cache/conftool/dbconfig/20230206-112839-root.json | |||
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43618 and previous config saved to /var/cache/conftool/dbconfig/20230206-112832-root.json | |||
* 11:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43617 and previous config saved to /var/cache/conftool/dbconfig/20230206-112825-root.json | |||
* 11:28 cgoubert@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on puppetmaster2002.codfw.wmnet,puppetmaster1002.eqiad.wmnet with reason: Decom | |||
* 11:27 cgoubert@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on puppetmaster2002.codfw.wmnet,puppetmaster1002.eqiad.wmnet with reason: Decom | |||
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43616 and previous config saved to /var/cache/conftool/dbconfig/20230206-111356-root.json | |||
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43615 and previous config saved to /var/cache/conftool/dbconfig/20230206-111351-root.json | |||
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43614 and previous config saved to /var/cache/conftool/dbconfig/20230206-111334-root.json | |||
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43613 and previous config saved to /var/cache/conftool/dbconfig/20230206-111327-root.json | |||
* 11:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43612 and previous config saved to /var/cache/conftool/dbconfig/20230206-111320-root.json | |||
* 11:03 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply | |||
* 11:03 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: apply | |||
* 11:03 akosiaris: deploy changeprop 0.10.19, adding wikivoyage to list of domains the mobile-sections get rerendered for. [[phab:T226931|T226931]] | |||
* 11:03 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply | |||
* 11:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: apply | |||
* 11:01 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: apply | |||
* 11:01 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply | |||
* 10:59 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply | |||
* 10:58 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply | |||
* 10:58 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: apply | |||
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43610 and previous config saved to /var/cache/conftool/dbconfig/20230206-105851-root.json | |||
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43609 and previous config saved to /var/cache/conftool/dbconfig/20230206-105846-root.json | |||
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43608 and previous config saved to /var/cache/conftool/dbconfig/20230206-105829-root.json | |||
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43607 and previous config saved to /var/cache/conftool/dbconfig/20230206-105822-root.json | |||
* 10:58 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43606 and previous config saved to /var/cache/conftool/dbconfig/20230206-105815-root.json | |||
* 10:56 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: apply | |||
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43605 and previous config saved to /var/cache/conftool/dbconfig/20230206-104346-root.json | |||
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43604 and previous config saved to /var/cache/conftool/dbconfig/20230206-104341-root.json | |||
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43603 and previous config saved to /var/cache/conftool/dbconfig/20230206-104324-root.json | |||
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43602 and previous config saved to /var/cache/conftool/dbconfig/20230206-104317-root.json | |||
* 10:43 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43601 and previous config saved to /var/cache/conftool/dbconfig/20230206-104310-root.json | |||
* 10:36 marostegui: Upgrade db1115 (db_inventory master) to 10.6. [[phab:T328408|T328408]] | |||
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43600 and previous config saved to /var/cache/conftool/dbconfig/20230206-102841-root.json | |||
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43599 and previous config saved to /var/cache/conftool/dbconfig/20230206-102837-root.json | |||
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43598 and previous config saved to /var/cache/conftool/dbconfig/20230206-102820-root.json | |||
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43597 and previous config saved to /var/cache/conftool/dbconfig/20230206-102812-root.json | |||
* 10:28 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43596 and previous config saved to /var/cache/conftool/dbconfig/20230206-102806-root.json | |||
* 10:27 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 10:27 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 10:26 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 10:23 cmooney@cumin1001: START - Cookbook sre.dns.netbox | |||
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2028 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43595 and previous config saved to /var/cache/conftool/dbconfig/20230206-101336-root.json | |||
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2027 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43594 and previous config saved to /var/cache/conftool/dbconfig/20230206-101332-root.json | |||
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43593 and previous config saved to /var/cache/conftool/dbconfig/20230206-101315-root.json | |||
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2024 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43592 and previous config saved to /var/cache/conftool/dbconfig/20230206-101308-root.json | |||
* 10:13 marostegui@cumin1001: dbctl commit (dc=all): 'es2020 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43591 and previous config saved to /var/cache/conftool/dbconfig/20230206-101301-root.json | |||
* 10:10 hashar@deploy1002: Finished deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided) (duration: 00m 38s) | |||
* 10:09 hashar@deploy1002: Started deploy [releng/jenkins-deploy@b798462] (releasing): (no justification provided) | |||
* 09:05 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | |||
* 09:05 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:886105{{!}}Fix and add mising parser test for maplink with suppressed text="" (T328739)]] (duration: 18m 56s) | |||
* 09:05 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | |||
* 09:04 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . | |||
* 09:04 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . | |||
* 08:56 urbanecm@deploy1002: wmde-fisch and urbanecm: Backport for [[gerrit:886105{{!}}Fix and add mising parser test for maplink with suppressed text="" (T328739)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* 08:46 urbanecm@deploy1002: Started scap: Backport for [[gerrit:886105{{!}}Fix and add mising parser test for maplink with suppressed text="" (T328739)]] | |||
* 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2094 db2097 db2103 db2104 db2105 db2106 db2121 db2122 db2132 db2133 db2136 db2142 db2145 db2146 db2153 db2154 db2155 db2156 db2157 db2158 db2175 db2176 db2183 [[phab:T327925|T327925]]', diff saved to https://phabricator.wikimedia.org/P43587 and previous config saved to /var/cache/conftool/dbconfig/20230206-073015-root.json | |||
* 07:19 marostegui@cumin1001: dbctl commit (dc=all): 'Depool es2020 es2024 es2026 es2027 es2028 [[phab:T327925|T327925]]', diff saved to https://phabricator.wikimedia.org/P43586 and previous config saved to /var/cache/conftool/dbconfig/20230206-071913-root.json | |||
* 07:17 hashar: Restarted Gerrit for deployment | |||
* 07:14 hashar@deploy1002: Finished deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json (duration: 00m 05s) | |||
* 07:14 hashar@deploy1002: Started deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json | |||
* 07:07 hashar@deploy1002: Finished deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json {{!}} [[phab:T328134|T328134]] (duration: 00m 10s) | |||
* 07:06 hashar@deploy1002: Started deploy [gerrit/gerrit@e09efc0]: remove plugins/.eslintrc.json {{!}} [[phab:T328134|T328134]] | |||
== | == 2023-02-05 == | ||
* | * 22:28 topranks: Re-enabling peering to Seabone/Telecom Italit AS 6762 on cr2-esams at AMS-IX | ||
* | * 14:39 cdanis: silenced NELHigh alert for 20 hours: Telecom Italy issues; alertmanager silence id 3fb3b999-9756-44af-a1e8-{{Gerrit|fd1faae8b9bf}} | ||
* 11:49 topranks: Manually deactivating peering to Telecom Italia / Seabone at AMS-IX on cr2-esams as they are having issues | |||
== | == 2023-02-03 == | ||
* | * 21:05 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* | * 21:04 cmooney@cumin1001: START - Cookbook sre.dns.netbox | ||
* 20: | * 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* 20: | * 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | ||
* | * 21:02 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | ||
* | * 21:00 cmooney@cumin1001: START - Cookbook sre.dns.netbox | ||
* 19: | * 20:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) | ||
* | * 20:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox | ||
* | * 19:44 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet | ||
* | * 19:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1090.eqiad.wmnet with OS bullseye | ||
* | * 19:00 dzahn@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "test what is not synced - dzahn@cumin2002" | ||
* | * 18:59 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test what is not synced - dzahn@cumin2002" | ||
* | * 18:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage | ||
* | * 18:49 topranks: Enabling 4x10G channelization for pic 0 QSFP 4 on cr1-codfw | ||
* | * 18:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage | ||
* | * 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1090.eqiad.wmnet with OS bullseye | ||
* | * 18:23 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet | ||
* | * 18:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS bullseye | ||
* | * 17:57 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage | ||
* | * 17:57 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage | ||
* | * 17:39 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet | ||
* | * 17:36 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1089.eqiad.wmnet with OS bullseye | ||
* | * 17:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1088.eqiad.wmnet with OS bullseye | ||
* | * 17:34 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet | ||
* | * 17:34 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1086.eqiad.wmnet with OS bullseye | ||
* | * 17:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage | ||
* | * 17:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage | ||
* 02: | * 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage | ||
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage | |||
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1086.eqiad.wmnet with OS bullseye | |||
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1089.eqiad.wmnet with OS bullseye | |||
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 16:44 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 16:41 cmooney@cumin1001: START - Cookbook sre.dns.netbox | |||
* 16:32 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2012.codfw.wmnet | |||
* 16:25 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2012.codfw.wmnet | |||
* 15:51 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): test (duration: 00m 26s) | |||
* 15:51 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): test | |||
* 15:23 milimetric@deploy1002: Finished deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection (duration: 00m 15s) | |||
* 15:22 milimetric@deploy1002: Started deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection | |||
* 14:31 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 09s) | |||
* 14:31 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) | |||
* 14:20 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2011.codfw.wmnet | |||
* 14:19 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 23s) | |||
* 14:18 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) | |||
* 14:13 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2011.codfw.wmnet | |||
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=ats-be | |||
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=cdn | |||
* 13:51 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1087.eqiad.wmnet with OS bullseye | |||
* 13:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage | |||
* 13:25 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage | |||
* 13:05 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1087.eqiad.wmnet with OS bullseye | |||
* 12:09 moritzm: installing node-moment security updates | |||
* 12:01 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 13s) | |||
* 12:00 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) | |||
* 11:58 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2010.codfw.wmnet | |||
* 11:58 moritzm: installing node-qs security updates | |||
* 11:50 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2010.codfw.wmnet | |||
* 11:35 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2009.codfw.wmnet | |||
* 11:28 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2009.codfw.wmnet | |||
* 10:44 moritzm: updating perf on buster hosts | |||
* 10:24 stevemunene@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:11 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2008.codfw.wmnet | |||
* 10:07 stevemunene@cumin1001: END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:06 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:03 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2008.codfw.wmnet | |||
* 09:51 moritzm: installing ruby-rack security updates | |||
* 09:31 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:31 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:24 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:23 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:23 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:19 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet | |||
* 09:14 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet | |||
* 09:13 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:13 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:07 moritzm: installing modsecurity-crs security updates | |||
* 09:02 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet | |||
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet | |||
* 05:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1084.eqiad.wmnet with OS bullseye | |||
* 05:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1085.eqiad.wmnet with OS bullseye | |||
* 04:50 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage | |||
* 04:47 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage | |||
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage | |||
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage | |||
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1084.eqiad.wmnet with OS bullseye | |||
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1085.eqiad.wmnet with OS bullseye | |||
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet | |||
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet | |||
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1083.eqiad.wmnet with OS bullseye | |||
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1082.eqiad.wmnet with OS bullseye | |||
* 03:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage | |||
* 03:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage | |||
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage | |||
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage | |||
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1082.eqiad.wmnet with OS bullseye | |||
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1083.eqiad.wmnet with OS bullseye | |||
* 03:20 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet | |||
* 03:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1080.eqiad.wmnet with OS bullseye | |||
* 02:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage | |||
* 02:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage | |||
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=ats-be | |||
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=cdn | |||
* 02:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1081.eqiad.wmnet with OS bullseye | |||
* 02:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye | |||
* 02:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage | |||
* 02:00 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage | |||
* 01:38 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS bullseye | |||
* 01:31 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye | |||
* 00:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye | |||
== | == 2023-02-02 == | ||
* | * 22:58 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye | ||
* | * 22:15 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet | ||
* | * 22:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS bullseye | ||
* | * 22:01 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye | ||
* | * 22:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet | ||
* | * 21:58 zabe@deploy1002: Finished scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] (duration: 07m 58s) | ||
* | * 21:52 zabe@deploy1002: zabe: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | ||
* | * 21:50 zabe@deploy1002: Started scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] | ||
* | * 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1078.eqiad.wmnet with OS bullseye | ||
* | * 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage | ||
* 23 | * 21:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage | ||
* | * 21:30 brennen: end of utc late backport & config window | ||
* | * 21:30 brennen@deploy1002: Finished scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] (duration: 11m 14s) | ||
* 21:23 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage | |||
* | * 21:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS bullseye | ||
* 21:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet | |||
* 21: | * 21:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1077.eqiad.wmnet with OS bullseye | ||
* 21:21 brennen@deploy1002: brennen and nray: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* | * 21:20 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage | ||
* | * 21:19 brennen@deploy1002: Started scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] | ||
* | * 21:18 brennen@deploy1002: Finished scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] (duration: 12m 02s) | ||
* 20: | * 21:07 brennen@deploy1002: brennen and dreamyjazz: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet | ||
* 20: | * 21:06 brennen@deploy1002: Started scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] | ||
* 20: | * 20:59 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye | ||
* 20:59 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1078.eqiad.wmnet with OS bullseye | |||
* 20: | * 20:52 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage | ||
* | * 20:49 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage | ||
* | * 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye | ||
* | * 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1077.eqiad.wmnet with OS bullseye | ||
* | * 20:23 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include bullseye-wikimedia /home/rzl/httpbb/bullseye/httpbb_0.0.3-1+deb11u1_amd64.changes # [[phab:T328280|T328280]] | ||
* | * 20:21 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include buster-wikimedia /home/rzl/httpbb/buster/httpbb_0.0.3-1_amd64.changes # [[phab:T328280|T328280]] | ||
* | * 20:11 zabe@deploy1002: Finished scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] (duration: 09m 39s) | ||
* | * 20:03 zabe@deploy1002: zabe: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet | ||
* | * 20:02 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host elastic2037.codfw.wmnet | ||
* 19: | * 20:01 zabe@deploy1002: Started scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] | ||
* 19: | * 19:55 bking@cumin1001: START - Cookbook sre.hosts.reboot-single for host elastic2037.codfw.wmnet | ||
* | * 19:54 ryankemper: [[phab:T328674|T328674]] [Elastic] With puppet disabled on elastic* fleet, `ryankemper@elastic2037:~$ sudo run-puppet-agent --force` to verify changes in https://gerrit.wikimedia.org/r/886055 | ||
* | * 19:30 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.21 refs [[phab:T325584|T325584]] | ||
* | * 19:28 zabe@deploy1002: say aborted: (duration: 00m 03s) | ||
* | * 18:42 zabe@deploy1002: Finished scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] (duration: 08m 19s) | ||
* 18:36 zabe@deploy1002: zabe: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet | |||
* | * 18:34 zabe@deploy1002: Started scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] | ||
* | * 18:08 aokoth@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0 | ||
* | * 18:08 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply | ||
* | * 18:08 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply | ||
* | * 18:08 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2043.codfw.wmnet with OS bullseye | ||
* | * 18:07 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply | ||
* | * 18:06 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply | ||
* | * 18:05 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply | ||
* | * 18:05 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply | ||
* | * 18:03 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1037.eqiad.wmnet with OS bullseye | ||
* | * 17:52 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage | ||
* | * 17:49 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage | ||
* | * 17:47 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage | ||
* | * 17:45 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage | ||
* | * 17:33 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2043.codfw.wmnet with OS bullseye | ||
* | * 17:32 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1037.eqiad.wmnet with OS bullseye | ||
* 17:29 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0 | |||
* | * 17:12 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync | ||
* | * 17:12 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: sync | ||
* | * 16:53 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply | ||
* | * 16:52 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply | ||
* | * 16:51 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply | ||
* | * 16:50 dancy@deploy1002: Installation of scap version "4.34.0" completed for 561 hosts | ||
* | * 16:50 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply | ||
* | * 16:50 dancy@deploy1002: Installing scap version "4.34.0" for 561 hosts | ||
* | * 16:50 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 16:49 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 16:48 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 16:48 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* 16:47 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: sync | |||
* 16:46 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: sync | |||
* | * 16:25 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2007.codfw.wmnet | ||
* | * 16:18 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply | ||
* | * 16:17 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply | ||
* | * 16:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2007.codfw.wmnet | ||
* | * 16:17 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply | ||
* | * 16:16 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply | ||
* | * 16:16 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 16:15 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 16:10 volans: uploaded python3-wmflib_1.2.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia | ||
* | * 16:10 dzahn@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0 | ||
* | * 15:40 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided) (duration: 07m 01s) | ||
* | * 15:38 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:37 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:35 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:35 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:34 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0 | ||
* | * 15:33 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided) | ||
* | * 15:24 jmm@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti3004 | ||
* | * 15:17 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti3004 | ||
* | * 15:06 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2006.codfw.wmnet | ||
* | * 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* | * 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002" | ||
* | * 15:02 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002" | ||
* | * 15:00 vgutierrez: rolling restart of varnish in cache::text - [[phab:T315676|T315676]] | ||
* | * 14:59 jmm@cumin2002: START - Cookbook sre.dns.netbox | ||
* | * 14:59 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2006.codfw.wmnet | ||
* | * 14:55 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* 14:45 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | |||
* | * 14:39 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* 14:31 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2005.codfw.wmnet | |||
* | * 14:29 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* 14:25 moritzm: installing containerd security updates on codfw k8s nodes | |||
* | * 14:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2005.codfw.wmnet | ||
* 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=ats-be | |||
* | * 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=cdn | ||
* | * 13:10 kharlan:: Deployed security patch for [[phab:T328643|T328643]] | ||
* 13:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1076.eqiad.wmnet with OS bullseye | |||
* | * 13:04 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . | ||
* | * 13:03 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . | ||
* 08 | * 13:03 kharlan:: Deployed security patch for [[phab:T328643|T328643]] | ||
* 13:02 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . | |||
* 08 | * 13:01 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2004.codfw.wmnet | ||
* | * 13:00 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . | ||
* | * 12:55 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2004.codfw.wmnet | ||
* | * 12:47 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage | ||
* | * 12:47 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . | ||
* | * 12:46 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . | ||
* | * 12:44 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage | ||
* | * 12:42 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | ||
* 12:42 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | |||
* | * 12:39 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . | ||
* 12:39 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . | |||
* 12:29 btullis@deploy1002: Finished deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade (duration: 00m 42s) | |||
* | * 12:29 claime: Work ongoing on m2 and m3 | ||
* 12:29 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2003.codfw.wmnet | |||
* | * 12:29 btullis@deploy1002: Started deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade | ||
* 12:23 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1076.eqiad.wmnet with OS bullseye | |||
* 12:22 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2003.codfw.wmnet | |||
* 12:08 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . | |||
* 12:08 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . | |||
* 11:46 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . | |||
* 11:42 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply | |||
* 11:42 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply | |||
* 11:41 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply | |||
* 11:41 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply | |||
* 11:40 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply | |||
* 11:39 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply | |||
* 11:38 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply | |||
* 11:37 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply | |||
* 11:37 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-4.out # [[phab:T328634|T328634]] – made some progress then errored out again | |||
* 11:32 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-3.out # [[phab:T328634|T328634]] – seemed to finish the first 20 pages and then go into an infinite loop, I Ctrl+Ced it | |||
* 11:28 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-2.out # [[phab:T328634|T328634]] – another error but made more progress | |||
* 11:23 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes.out # [[phab:T328634|T328634]] – failed quickly, details in task | |||
* 11:22 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: sync | |||
* 11:22 elukey@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: sync | |||
* 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | |||
* 11:02 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | |||
* 10:27 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2002.codfw.wmnet | |||
* 10:19 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2002.codfw.wmnet | |||
* 10:17 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . | |||
* 10:11 moritzm: restarting FPM on mw canaries to pick up tiff security updates | |||
* 10:04 moritzm: installing tiff security updates | |||
* 09:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2001.codfw.wmnet | |||
* 09:55 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync | |||
* 09:54 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync | |||
* 09:51 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2001.codfw.wmnet | |||
* 09:40 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync | |||
* 09:40 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: sync | |||
* 09:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 398143 | |||
* 09:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 398143 | |||
* 09:16 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6 | |||
* 09:13 apergos: UTC morning backport and config training window done | |||
* 09:13 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync | |||
* 09:12 elukey@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-main: sync | |||
* 09:11 elukey: roll restart of eventgate-main pods in wikikube eqiad/codfw to pick up new stream configs - [[phab:T328576|T328576]] | |||
* 08:57 ariel@deploy1002: Finished scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] (duration: 10m 56s) | |||
* 08:48 ariel@deploy1002: ariel and aishik: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* 08:46 ariel@deploy1002: Started scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] | |||
* 08:39 jelto@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6 | |||
* 08:37 tgr@deploy1002: Finished scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] (duration: 14m 26s) | |||
* 08:27 tgr@deploy1002: tgr: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | |||
* 08:23 tgr@deploy1002: Started scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] | |||
* 06:17 kart_: Updated cxserver to 2023-02-02-004918-production ([[phab:T129470|T129470]], [[phab:T172035|T172035]], [[phab:T327842|T327842]]) | |||
* 06:16 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply | |||
* 06:15 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply | |||
* 06:13 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply | |||
* 06:12 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply | |||
* 06:09 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply | |||
* 06:09 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply | |||
* 04:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5024.eqsin.wmnet | |||
* 03:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS bullseye | |||
* 03:21 ejegg: payments-wiki upgraded from {{Gerrit|f20a2208}} to {{Gerrit|53d1a58d}} | |||
* 02:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage | |||
* 02:46 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage | |||
* 02:14 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye | |||
* 02:14 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5024.eqsin.wmnet with OS bullseye | |||
* 01:56 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye | |||
* 01:55 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet | |||
* 01:55 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5023.eqsin.wmnet with OS bullseye | |||
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be | |||
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=cdn | |||
* 01:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1075.eqiad.wmnet with OS bullseye | |||
* 01:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage | |||
* 01:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage | |||
* 01:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage | |||
* 01:18 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage | |||
* 01:07 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1075.eqiad.wmnet with OS bullseye | |||
* 00:44 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5023.eqsin.wmnet with OS bullseye | |||
* 00:06 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5022.eqsin.wmnet | |||
* 00:04 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS bullseye | |||
== | == 2023-02-01 == | ||
* 23: | * 23:45 zabe@deploy1002: Finished scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] (duration: 08m 07s) | ||
* 23: | * 23:39 zabe@deploy1002: zabe: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet | ||
* 23: | * 23:37 zabe@deploy1002: Started scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] | ||
* 23: | * 23:31 rzl@cumin2002: dbctl commit (dc=all): 'Depool db2181', diff saved to https://phabricator.wikimedia.org/P43574 and previous config saved to /var/cache/conftool/dbconfig/20230201-233140-rzl.json | ||
* 23: | * 23:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage | ||
* 23: | * 23:27 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage | ||
* 23: | * 23:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: security release | ||
* 23: | * 23:17 dancy@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.21 refs [[phab:T325584|T325584]] (duration: 06m 57s) | ||
* | * 23:10 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.21 refs [[phab:T325584|T325584]] | ||
* | * 23:01 zabe@deploy1002: Finished scap: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] (duration: 07m 45s) | ||
* 22:55 zabe@deploy1002: zabe: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet | |||
* | * 22:54 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS bullseye | ||
* 22:53 zabe@deploy1002: Started scap: Backport for [[gerrit:885781{{!}}CachingKartographerEmbeddingHandler: Fall back to Special:BlankPage title (T328601)]] | |||
* | * 22:49 zabe@deploy1002: Finished scap: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] (duration: 13m 03s) | ||
* | * 22:47 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release | ||
* | * 22:40 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS bullseye | ||
* 22:38 zabe@deploy1002: zabe: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet | |||
* | * 22:36 zabe@deploy1002: Started scap: Backport for [[gerrit:885898{{!}}Stop writing to cuc_comment_id in group0 wikis (T233004)]] | ||
* | * 22:32 kindrobot: close UTC late backport window | ||
* 22:31 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] (duration: 10m 37s) | |||
* 22:22 kindrobot@deploy1002: nray and kindrobot: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* | * 22:21 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885841{{!}}Enable client preferences for group1 (T327979)]] | ||
* | * 22:14 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] (duration: 18m 14s) | ||
* | * 21:57 kindrobot@deploy1002: kindrobot and sbailey: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet | ||
* | * 21:57 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore100*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001 | ||
* | * 21:56 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885852{{!}}Enable Linter write namespace, tag and template for all wikis (T299612)]] | ||
* | * 21:53 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 21:52 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] (duration: 14m 53s) | ||
* | * 21:43 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS bullseye | ||
* | * 21:39 eevans@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore100*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001 | ||
* | * 21:39 kindrobot@deploy1002: dreamyjazz and kindrobot: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet | ||
* | * 21:37 kindrobot@deploy1002: Started scap: Backport for [[gerrit:885358{{!}}Disable write old for CheckUserLog reason on group 0 (T233004)]] | ||
* 21:32 kindrobot@deploy1002: Finished scap: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] (duration: 13m 56s) | |||
* | * 21:26 eevans@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=sessionstore,name=codfw | ||
* | * 21:26 eevans@puppetmaster1001: conftool action : get/pooled=true; selector: dnsdisc=sessionstore,name=codfw | ||
* 21:26 eevans@puppetmaster1001: conftool action : get/pooled=true; selector: dnsdisc=sessionstore,name=codfw | |||
* | * 21:24 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 21:20 kindrobot@deploy1002: arlolra and kindrobot: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet | ||
* | * 21:19 eevans@cumin1001: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001 | ||
* | * 21:18 kindrobot@deploy1002: Started scap: Backport for [[gerrit:865214{{!}}Disable wgParserEnableLegacyMediaDOM on group1 wikis (T314318)]] | ||
* | * 21:14 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3065.esams.wmnet | ||
* | * 21:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3065.esams.wmnet with OS bullseye | ||
* | * 21:03 kindrobot: start UTC late backport deployment window | ||
* | * 21:02 eevans@cumin1001: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200*: Applying new TLS certificates — [[phab:T327675|T327675]] - eevans@cumin1001 | ||
* | * 20:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3065.esams.wmnet with reason: host reimage | ||
* | * 20:44 eevans@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=sessionstore,name=codfw | ||
* 14: | * 20:43 urandom: depooling sessionstore —codfw— in preparation for Cassandra restarts — [[phab:T327675|T327675]] | ||
* 20:42 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3065.esams.wmnet with reason: host reimage | |||
* | * 20:40 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3064.esams.wmnet | ||
* | * 20:38 eevans@puppetmaster1001: conftool action : get/pooled; selector: dnsdisc=$SERVICE,name=$DC | ||
* | * 20:33 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3064.esams.wmnet with OS bullseye | ||
* 20:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3065.esams.wmnet with OS bullseye | |||
* | * 20:21 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3063.esams.wmnet | ||
* 20:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3064.esams.wmnet with reason: host reimage | |||
* | * 20:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3063.esams.wmnet with OS bullseye | ||
* | * 20:08 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3064.esams.wmnet with reason: host reimage | ||
* | * 20:03 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5031.eqsin.wmnet,service=ats-be | ||
* | * 20:03 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5031.eqsin.wmnet,service=cdn | ||
* | * 20:00 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5031.eqsin.wmnet with OS bullseye | ||
* | * 19:53 dancy: The train is blocked on [[phab:T328601|T328601]] | ||
* | * 19:49 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3064.esams.wmnet with OS bullseye | ||
* | * 19:49 dancy@deploy1002: Synchronized php: group1 wikis to 1.40.0-wmf.20 refs [[phab:T325584|T325584]] (duration: 06m 36s) | ||
* | * 19:49 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3062.esams.wmnet | ||
* 19:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3062.esams.wmnet with OS bullseye | |||
* 19:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3063.esams.wmnet with reason: host reimage | |||
* | * 19:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3063.esams.wmnet with reason: host reimage | ||
* | * 19:42 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.20 refs [[phab:T325584|T325584]] | ||
* 19:41 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5021.eqsin.wmnet,service=ats-be | |||
* | * 19:41 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5021.eqsin.wmnet,service=cdn | ||
* 19:37 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5021.eqsin.wmnet with OS bullseye | |||
* | * 19:33 dancy@deploy1002: deploy-promote aborted: (duration: 11m 58s) | ||
* | * 19:33 dancy@deploy1002: sync-file aborted: group1 wikis to 1.40.0-wmf.21 refs [[phab:T325584|T325584]] (duration: 03m 38s) | ||
* | * 19:30 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5031.eqsin.wmnet with reason: host reimage | ||
* | * 19:29 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.21 refs [[phab:T325584|T325584]] | ||
* | * 19:27 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5031.eqsin.wmnet with reason: host reimage | ||
* | * 19:26 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3062.esams.wmnet with reason: host reimage | ||
* | * 19:24 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3063.esams.wmnet with OS bullseye | ||
* 19:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3061.esams.wmnet | |||
* | * 19:24 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3062.esams.wmnet with reason: host reimage | ||
* | * 19:17 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3061.esams.wmnet with OS bullseye | ||
* | * 19:04 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5021.eqsin.wmnet with reason: host reimage | ||
* 19:03 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3062.esams.wmnet with OS bullseye | |||
* 19:02 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3060.esams.wmnet | |||
* | * 19:02 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3060.esams.wmnet with OS bullseye | ||
* 19:01 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5021.eqsin.wmnet with reason: host reimage | |||
* | * 18:56 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3061.esams.wmnet with reason: host reimage | ||
* | * 18:55 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye | ||
* | * 18:55 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5031.eqsin.wmnet with OS bullseye | ||
* | * 18:52 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3061.esams.wmnet with reason: host reimage | ||
* | * 18:47 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye | ||
* | * 18:46 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5031.eqsin.wmnet with OS bullseye | ||
* | * 18:39 jbond@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts puppetmaster2003.codfw.wmnet | ||
* | * 18:38 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3060.esams.wmnet with reason: host reimage | ||
* | * 18:37 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5031.eqsin.wmnet with OS bullseye | ||
* | * 18:35 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3060.esams.wmnet with reason: host reimage | ||
* | * 18:32 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3061.esams.wmnet with OS bullseye | ||
* | * 18:31 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3059.esams.wmnet | ||
* | * 18:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3059.esams.wmnet with OS bullseye | ||
* | * 18:29 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye | ||
* | * 18:29 jbond@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts puppetmaster2003.codfw.wmnet | ||
* | * 18:29 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5021.eqsin.wmnet with OS bullseye | ||
* | * 18:22 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye | ||
* | * 18:21 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp1075.eqiad.wmnet with reason: downtimed for idrac firmware testing | ||
* | * 18:20 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on cp1075.eqiad.wmnet with reason: downtimed for idrac firmware testing | ||
* 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet,service=ats-be | |||
* | * 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5030.eqsin.wmnet,service=cdn | ||
* | * 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet,service=ats-be | ||
* | * 18:19 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet,service=cdn | ||
* | * 18:13 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3060.esams.wmnet with OS bullseye | ||
* | * 18:13 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3058.esams.wmnet | ||
* 18:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3058.esams.wmnet with OS bullseye | |||
* | * 18:10 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5030.eqsin.wmnet with OS bullseye | ||
* | * 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43573 and previous config saved to /var/cache/conftool/dbconfig/20230201-181036-root.json | ||
* | * 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43572 and previous config saved to /var/cache/conftool/dbconfig/20230201-181031-root.json | ||
* 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43571 and previous config saved to /var/cache/conftool/dbconfig/20230201-181024-root.json | |||
* | * 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43570 and previous config saved to /var/cache/conftool/dbconfig/20230201-181016-root.json | ||
* | * 18:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43569 and previous config saved to /var/cache/conftool/dbconfig/20230201-181011-root.json | ||
* | * 18:06 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3059.esams.wmnet with reason: host reimage | ||
* 18:03 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3059.esams.wmnet with reason: host reimage | |||
* | * 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43568 and previous config saved to /var/cache/conftool/dbconfig/20230201-175531-root.json | ||
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43567 and previous config saved to /var/cache/conftool/dbconfig/20230201-175526-root.json | |||
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43566 and previous config saved to /var/cache/conftool/dbconfig/20230201-175519-root.json | |||
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43565 and previous config saved to /var/cache/conftool/dbconfig/20230201-175511-root.json | |||
* 17:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43564 and previous config saved to /var/cache/conftool/dbconfig/20230201-175506-root.json | |||
* 17:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P43563 and previous config saved to /var/cache/conftool/dbconfig/20230201-175446-root.json | |||
* 17:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3058.esams.wmnet with reason: host reimage | |||
* 17:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3058.esams.wmnet with reason: host reimage | |||
* 17:41 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3059.esams.wmnet with OS bullseye | |||
* 17:40 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3057.esams.wmnet | |||
* 17:40 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3057.esams.wmnet with OS bullseye | |||
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43562 and previous config saved to /var/cache/conftool/dbconfig/20230201-174026-root.json | |||
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43561 and previous config saved to /var/cache/conftool/dbconfig/20230201-174021-root.json | |||
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43560 and previous config saved to /var/cache/conftool/dbconfig/20230201-174015-root.json | |||
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43559 and previous config saved to /var/cache/conftool/dbconfig/20230201-174007-root.json | |||
* 17:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43558 and previous config saved to /var/cache/conftool/dbconfig/20230201-174001-root.json | |||
* 17:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P43557 and previous config saved to /var/cache/conftool/dbconfig/20230201-173941-root.json | |||
* 17:39 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage | |||
* 17:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5030.eqsin.wmnet with reason: host reimage | |||
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43555 and previous config saved to /var/cache/conftool/dbconfig/20230201-172521-root.json | |||
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43554 and previous config saved to /var/cache/conftool/dbconfig/20230201-172516-root.json | |||
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43553 and previous config saved to /var/cache/conftool/dbconfig/20230201-172510-root.json | |||
* 17:25 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43552 and previous config saved to /var/cache/conftool/dbconfig/20230201-172502-root.json | |||
* 17:24 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43551 and previous config saved to /var/cache/conftool/dbconfig/20230201-172456-root.json | |||
* 17:24 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P43550 and previous config saved to /var/cache/conftool/dbconfig/20230201-172436-root.json | |||
* 17:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3058.esams.wmnet with OS bullseye | |||
* 17:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3056.esams.wmnet | |||
* 17:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3056.esams.wmnet with OS bullseye | |||
* 17:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3057.esams.wmnet with reason: host reimage | |||
* 17:17 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5019.eqsin.wmnet with OS bullseye | |||
* 17:15 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3057.esams.wmnet with reason: host reimage | |||
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43549 and previous config saved to /var/cache/conftool/dbconfig/20230201-171016-root.json | |||
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43548 and previous config saved to /var/cache/conftool/dbconfig/20230201-171011-root.json | |||
* 17:10 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43547 and previous config saved to /var/cache/conftool/dbconfig/20230201-171005-root.json | |||
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43546 and previous config saved to /var/cache/conftool/dbconfig/20230201-170957-root.json | |||
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43545 and previous config saved to /var/cache/conftool/dbconfig/20230201-170951-root.json | |||
* 17:09 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P43544 and previous config saved to /var/cache/conftool/dbconfig/20230201-170931-root.json | |||
* 16:57 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:57 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:57 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3056.esams.wmnet with reason: host reimage | |||
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43543 and previous config saved to /var/cache/conftool/dbconfig/20230201-165512-root.json | |||
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43542 and previous config saved to /var/cache/conftool/dbconfig/20230201-165506-root.json | |||
* 16:55 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43541 and previous config saved to /var/cache/conftool/dbconfig/20230201-165500-root.json | |||
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43540 and previous config saved to /var/cache/conftool/dbconfig/20230201-165452-root.json | |||
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3056.esams.wmnet with reason: host reimage | |||
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43539 and previous config saved to /var/cache/conftool/dbconfig/20230201-165446-root.json | |||
* 16:54 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3057.esams.wmnet with OS bullseye | |||
* 16:54 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43538 and previous config saved to /var/cache/conftool/dbconfig/20230201-165426-root.json | |||
* 16:42 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:42 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:40 marostegui@cumin1001: dbctl commit (dc=all): 'es2026 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43536 and previous config saved to /var/cache/conftool/dbconfig/20230201-164007-root.json | |||
* 16:40 marostegui@cumin1001: dbctl commit (dc=all): 'db2158 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43535 and previous config saved to /var/cache/conftool/dbconfig/20230201-164002-root.json | |||
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2157 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43534 and previous config saved to /var/cache/conftool/dbconfig/20230201-163955-root.json | |||
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2146 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43533 and previous config saved to /var/cache/conftool/dbconfig/20230201-163947-root.json | |||
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2136 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43532 and previous config saved to /var/cache/conftool/dbconfig/20230201-163941-root.json | |||
* 16:39 marostegui@cumin1001: dbctl commit (dc=all): 'db2106 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43531 and previous config saved to /var/cache/conftool/dbconfig/20230201-163921-root.json | |||
* 16:33 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:33 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp3056.esams.wmnet with OS bullseye | |||
* 16:31 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:29 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage | |||
* 16:26 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage | |||
* 16:25 jynus: reloaded apache on mailman | |||
* 16:25 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5030.eqsin.wmnet with OS bullseye | |||
* 16:23 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . | |||
* 16:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . | |||
* 16:15 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply | |||
* 16:14 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply | |||
* 16:14 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply | |||
* 16:13 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply | |||
* 15:53 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye | |||
* 15:51 sukhe@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5019.eqsin.wmnet with OS bullseye | |||
* 15:31 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye | |||
* 14:56 sukhe: cp1075.eqiad.wmnet for idrac firmware upgrade testing | |||
* 14:55 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=ats-be | |||
* 14:55 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=cdn | |||
* 14:52 awight: EU deployment window complete | |||
* 14:48 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. | |||
* 14:48 awight@deploy1002: Finished scap: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] (duration: 08m 25s) | |||
* 14:47 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'. | |||
* 14:41 awight@deploy1002: elukey and awight: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet | |||
* 14:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2136 db2158 db2157 es2026 db2106 db2146 [[phab:T327404|T327404]]', diff saved to https://phabricator.wikimedia.org/P43530 and previous config saved to /var/cache/conftool/dbconfig/20230201-144152-root.json | |||
* 14:40 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. | |||
* 14:40 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'. | |||
* 14:40 awight@deploy1002: Started scap: Backport for [[gerrit:884155{{!}}wmf-config: add new revision-score streams for EventGate main (T317768)]] | |||
* 14:39 ayounsi@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. | |||
* 14:39 ayounsi@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'. | |||
* 14:37 awight@deploy1002: Finished scap: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] (duration: 09m 22s) | |||
* 14:29 awight@deploy1002: jdrewniak and awight: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet | |||
* 14:28 awight@deploy1002: Started scap: Backport for [[gerrit:885391{{!}}Add cswiki to desktop-improvements group. (T328154)]] | |||
* 14:26 awight@deploy1002: Finished scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] (duration: 09m 07s) | |||
* 14:19 awight@deploy1002: awight and mlitn: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet | |||
* 14:17 awight@deploy1002: Started scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] | |||
* 14:11 awight@deploy1002: backport aborted: (duration: 06m 09s) | |||
* 14:11 awight@deploy1002: sync-world aborted: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] (duration: 03m 36s) | |||
* 14:09 awight@deploy1002: mlitn and awight: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | |||
* 14:07 awight@deploy1002: Started scap: Backport for [[gerrit:885798{{!}}Squashed diff to catch up to master]] | |||
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast3005.wikimedia.org | |||
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3005.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 14:06 moritzm: updating perf on Bullseye hosts | |||
* 14:05 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3005.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 13:55 jmm@cumin2002: START - Cookbook sre.dns.netbox | |||
* 13:51 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast3005.wikimedia.org | |||
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5002.wikimedia.org | |||
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 13:47 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5002.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 13:43 jmm@cumin2002: START - Cookbook sre.dns.netbox | |||
* 13:36 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts bast5002.wikimedia.org | |||
* 13:21 moritzm: installing curl security updates on bullseye | |||
* 13:00 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main | |||
* 12:59 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main | |||
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2003.codfw.wmnet | |||
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 12:41 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 12:40 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 12:31 jmm@cumin2002: START - Cookbook sre.dns.netbox | |||
* 12:27 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2003.codfw.wmnet | |||
* 12:16 jmm@cumin2002: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for testvm2002.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 | |||
* 12:15 jmm@cumin2002: START - Cookbook sre.puppet.renew-cert for testvm2002.codfw.wmnet: Renew puppet certificate - jmm@cumin2002 | |||
* 11:29 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part III ([[phab:T308932|T308932]]) (duration: 06m 43s) | |||
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2001.codfw.wmnet | |||
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 11:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 11:24 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" | |||
* 11:22 jgiannelos@deploy1002: Finished deploy [kartotherian/deploy@e1ca693] (codfw): Allow stylesheets through CSP (duration: 01m 45s) | |||
* 11:21 ladsgroup@deploy1002: Synchronized multiversion/MWConfigCacheGenerator.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part II ([[phab:T308932|T308932]]) (duration: 07m 04s) | |||
* 11:21 jmm@cumin2002: START - Cookbook sre.dns.netbox | |||
* 11:20 jgiannelos@deploy1002: Started deploy [kartotherian/deploy@e1ca693] (codfw): Allow stylesheets through CSP | |||
* 11:17 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts testvm2001.codfw.wmnet | |||
* 11:17 jgiannelos@deploy1002: Finished deploy [kartotherian/deploy@e1ca693] (eqiad): Allow stylesheets through CSP (duration: 00m 51s) | |||
* 11:16 jgiannelos@deploy1002: Started deploy [kartotherian/deploy@e1ca693] (eqiad): Allow stylesheets through CSP | |||
* 11:14 ladsgroup@deploy1002: Synchronized wmf-config/ext-CirrusSearch.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part I ([[phab:T308932|T308932]]) (duration: 07m 04s) | |||
* 11:01 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a8840b0] (duration: 01m 18s) | |||
* 11:00 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a8840b0] | |||
* 10:59 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0] (thin): Regular analytics weekly train THIN [analytics/refinery@a8840b0] (duration: 00m 05s) | |||
* 10:59 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0] (thin): Regular analytics weekly train THIN [analytics/refinery@a8840b0] | |||
* 10:58 stevemunene@deploy1002: Finished deploy [analytics/refinery@a8840b0]: Regular analytics weekly train [analytics/refinery@a8840b0] (duration: 04m 29s) | |||
* 10:54 stevemunene@deploy1002: Started deploy [analytics/refinery@a8840b0]: Regular analytics weekly train [analytics/refinery@a8840b0] | |||
* 10:52 steve_munene: Deploying refinery for ops week | |||
* 10:42 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | |||
* 10:42 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | |||
* 10:42 zabe: start running migrateRevisionCommentTemp in remaining sections (for now except s3) in screens # [[phab:T275246|T275246]] | |||
* 10:42 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. | |||
* 10:42 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. | |||
* 10:41 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. | |||
* 10:41 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. | |||
* 10:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb2002.codfw.wmnet with OS bullseye | |||
* 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb2002.codfw.wmnet with reason: host reimage | |||
* 10:05 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on krb2002.codfw.wmnet with reason: host reimage | |||
* 10:01 godog: upgrade grafana to 8.5.20 on cloudmetrics* - [[phab:T328405|T328405]] | |||
* 09:57 godog: upgrade grafana to 8.5.20 on grafana1002 - [[phab:T328405|T328405]] | |||
* 09:50 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host krb2002.codfw.wmnet with OS bullseye | |||
* 09:47 godog: upgrade grafana to 8.5.20 on grafana2001 - [[phab:T328405|T328405]] | |||
* 09:15 urbanecm: Clean sign up throttle for IP 195.113.145.2 (via resetAuthenticationThrottle.php; [[phab:T328521|T328521]]) | |||
* 09:14 urbanecm@deploy1002: Finished scap: Backport for [[gerrit:885734{{!}}Add new throttle rule (T328521)]] (duration: 07m 24s) | |||
* 09:07 urbanecm@deploy1002: Started scap: Backport for [[gerrit:885734{{!}}Add new throttle rule (T328521)]] | |||
* 09:06 urbanecm@deploy1002: backport aborted: (duration: 00m 01s) | |||
* 09:05 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] (duration: 15m 06s) | |||
* 08:54 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: apply on main | |||
* 08:54 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main | |||
* 08:52 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet | |||
* 08:50 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:883620{{!}}Create additional namespaces on shn.wikibooks (T327850)]] | |||
* 08:49 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] (duration: 08m 05s) | |||
* 08:45 jayme@cumin1001: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=k8s-ingress-staging | |||
* 08:45 jayme@cumin1001: conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=k8s-ingress-staging | |||
* 08:42 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | |||
* 08:41 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:885321{{!}}Add a wordmark to trwiktionary (T328499)]] | |||
* 08:40 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] (duration: 12m 26s) | |||
* 08:29 ladsgroup@deploy1002: superpes and ladsgroup: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet | |||
* 08:27 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:884934{{!}}Add mobile wordmark to cswiktionary (T328357)]] | |||
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | |||
* 08:27 elukey@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | |||
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. | |||
* 08:27 elukey@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. | |||
* 08:27 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] (duration: 09m 42s) | |||
* 08:19 jayme@cumin1001: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 6 hosts | |||
* 08:19 jayme@cumin1001: START - Cookbook sre.hosts.remove-downtime for 6 hosts | |||
* 08:19 ladsgroup@deploy1002: ladsgroup and krinkle: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet | |||
* 08:17 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:879926{{!}}Remove former EventLogging streams for navtiming (T281103 T286703 T308621 T323623)]] | |||
* 08:14 ladsgroup@deploy1002: Finished scap: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] (duration: 10m 15s) | |||
* 08:06 ladsgroup@deploy1002: phedenskog and ladsgroup: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet | |||
* 08:05 moritzm: installing libarchive security updates | |||
* 08:04 ladsgroup@deploy1002: Started scap: Backport for [[gerrit:726854{{!}}Remove unused eventlogging_RUMSpeedIndex stream (T286700)]] | |||
* 08:01 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 55821 | |||
* 07:57 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 55821 | |||
* 07:33 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43524 and previous config saved to /var/cache/conftool/dbconfig/20230201-073348-ladsgroup.json | |||
* 07:18 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P43523 and previous config saved to /var/cache/conftool/dbconfig/20230201-071841-ladsgroup.json | |||
* 07:03 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P43522 and previous config saved to /var/cache/conftool/dbconfig/20230201-070335-ladsgroup.json | |||
* 06:48 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43521 and previous config saved to /var/cache/conftool/dbconfig/20230201-064828-ladsgroup.json | |||
* 06:45 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T310011|T310011]])', diff saved to https://phabricator.wikimedia.org/P43520 and previous config saved to /var/cache/conftool/dbconfig/20230201-064311-ladsgroup.json | |||
* 06:44 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance | |||
* 06:43 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance | |||
* 06:42 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance | |||
* 06:42 ladsgroup@cumin1001: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance | |||
* 00:38 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3055.esams.wmnet | |||
* 00:37 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3055.esams.wmnet with OS bullseye | |||
* 00:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3055.esams.wmnet with reason: host reimage | |||
* 00:12 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3055.esams.wmnet with reason: host reimage | |||
* 00:02 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp3054.esams.wmnet | |||
* 00:01 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3054.esams.wmnet with OS bullseye | |||
==Archives == | |||