You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Server Admin Log: Difference between revisions
Jump to navigation
Jump to search
imported>Stashbot (mutante: deploy1002 - rsyncing home dirs from deploy1001) |
imported>Stashbot (cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)) |
||
(641 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
== | == 2023-02-03 == | ||
* 00: | * 21:05 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* 21:04 cmooney@cumin1001: START - Cookbook sre.dns.netbox | |||
* 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 21:04 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 21:02 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 21:00 cmooney@cumin1001: START - Cookbook sre.dns.netbox | |||
* 20:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) | |||
* 20:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox | |||
* 19:44 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet | |||
* 19:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1090.eqiad.wmnet with OS bullseye | |||
* 19:00 dzahn@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "test what is not synced - dzahn@cumin2002" | |||
* 18:59 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "test what is not synced - dzahn@cumin2002" | |||
* 18:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage | |||
* 18:49 topranks: Enabling 4x10G channelization for pic 0 QSFP 4 on cr1-codfw | |||
* 18:45 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1090.eqiad.wmnet with reason: host reimage | |||
* 18:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1090.eqiad.wmnet with OS bullseye | |||
* 18:23 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet | |||
* 18:19 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1088.eqiad.wmnet with OS bullseye | |||
* 17:57 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage | |||
* 17:57 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1088.eqiad.wmnet with reason: host reimage | |||
* 17:39 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet | |||
* 17:36 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1089.eqiad.wmnet with OS bullseye | |||
* 17:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1088.eqiad.wmnet with OS bullseye | |||
* 17:34 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet | |||
* 17:34 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1086.eqiad.wmnet with OS bullseye | |||
* 17:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage | |||
* 17:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage | |||
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1089.eqiad.wmnet with reason: host reimage | |||
* 17:09 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1086.eqiad.wmnet with reason: host reimage | |||
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1086.eqiad.wmnet with OS bullseye | |||
* 16:47 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1089.eqiad.wmnet with OS bullseye | |||
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | |||
* 16:45 cmooney@cumin1001: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 16:44 cmooney@cumin1001: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for cloudsw1-b1-codfw mgmt IP. - cmooney@cumin1001" | |||
* 16:41 cmooney@cumin1001: START - Cookbook sre.dns.netbox | |||
* 16:32 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2012.codfw.wmnet | |||
* 16:25 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2012.codfw.wmnet | |||
* 15:51 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): test (duration: 00m 26s) | |||
* 15:51 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): test | |||
* 15:23 milimetric@deploy1002: Finished deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection (duration: 00m 15s) | |||
* 15:22 milimetric@deploy1002: Started deploy [airflow-dags/analytics@ec3e0de]: Hotfix disabling skein log collection | |||
* 14:31 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 09s) | |||
* 14:31 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) | |||
* 14:20 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2011.codfw.wmnet | |||
* 14:19 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 23s) | |||
* 14:18 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) | |||
* 14:13 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2011.codfw.wmnet | |||
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=ats-be | |||
* 13:55 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet,service=cdn | |||
* 13:51 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1087.eqiad.wmnet with OS bullseye | |||
* 13:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage | |||
* 13:25 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: host reimage | |||
* 13:05 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1087.eqiad.wmnet with OS bullseye | |||
* 12:09 moritzm: installing node-moment security updates | |||
* 12:01 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) (duration: 00m 13s) | |||
* 12:00 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@598ff3c] (releasing): (no justification provided) | |||
* 11:58 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2010.codfw.wmnet | |||
* 11:58 moritzm: installing node-qs security updates | |||
* 11:50 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2010.codfw.wmnet | |||
* 11:35 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2009.codfw.wmnet | |||
* 11:28 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2009.codfw.wmnet | |||
* 10:44 moritzm: updating perf on buster hosts | |||
* 10:24 stevemunene@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:11 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2008.codfw.wmnet | |||
* 10:07 stevemunene@cumin1001: END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:06 stevemunene@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. | |||
* 10:03 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2008.codfw.wmnet | |||
* 09:51 moritzm: installing ruby-rack security updates | |||
* 09:31 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:31 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:24 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:23 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:23 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:19 ariel@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1001.eqiad.wmnet | |||
* 09:14 ariel@cumin1001: START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet | |||
* 09:13 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:13 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 09:07 moritzm: installing modsecurity-crs security updates | |||
* 09:02 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'. | |||
* 09:02 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'. | |||
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet | |||
* 05:16 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet | |||
* 05:15 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1084.eqiad.wmnet with OS bullseye | |||
* 05:13 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1085.eqiad.wmnet with OS bullseye | |||
* 04:50 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage | |||
* 04:47 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage | |||
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1084.eqiad.wmnet with reason: host reimage | |||
* 04:47 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1085.eqiad.wmnet with reason: host reimage | |||
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1084.eqiad.wmnet with OS bullseye | |||
* 04:25 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1085.eqiad.wmnet with OS bullseye | |||
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet | |||
* 04:24 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet | |||
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1083.eqiad.wmnet with OS bullseye | |||
* 04:11 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1082.eqiad.wmnet with OS bullseye | |||
* 03:48 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage | |||
* 03:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage | |||
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1082.eqiad.wmnet with reason: host reimage | |||
* 03:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1083.eqiad.wmnet with reason: host reimage | |||
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1082.eqiad.wmnet with OS bullseye | |||
* 03:21 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1083.eqiad.wmnet with OS bullseye | |||
* 03:20 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet | |||
* 03:09 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1080.eqiad.wmnet with OS bullseye | |||
* 02:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage | |||
* 02:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1080.eqiad.wmnet with reason: host reimage | |||
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=ats-be | |||
* 02:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=cdn | |||
* 02:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1081.eqiad.wmnet with OS bullseye | |||
* 02:23 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye | |||
* 02:03 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage | |||
* 02:00 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1081.eqiad.wmnet with reason: host reimage | |||
* 01:38 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1081.eqiad.wmnet with OS bullseye | |||
* 01:31 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye | |||
* 00:35 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye | |||
== | == 2023-02-02 == | ||
* | * 22:58 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp1080.eqiad.wmnet with OS bullseye | ||
* 22:15 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet | |||
* | * 22:12 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1079.eqiad.wmnet with OS bullseye | ||
* | * 22:01 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1080.eqiad.wmnet with OS bullseye | ||
* | * 22:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet | ||
* | * 21:58 zabe@deploy1002: Finished scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] (duration: 07m 58s) | ||
* | * 21:52 zabe@deploy1002: zabe: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | ||
* 21:50 zabe@deploy1002: Started scap: Backport for [[gerrit:886149{{!}}Stop writing to cuc_comment everywhere (T233004)]] | |||
* | * 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1078.eqiad.wmnet with OS bullseye | ||
* 21:47 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage | |||
* | * 21:44 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1079.eqiad.wmnet with reason: host reimage | ||
* 21:30 brennen: end of utc late backport & config window | |||
* | * 21:30 brennen@deploy1002: Finished scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] (duration: 11m 14s) | ||
* 21:23 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage | |||
* | * 21:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1079.eqiad.wmnet with OS bullseye | ||
* | * 21:22 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet | ||
* 21:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1077.eqiad.wmnet with OS bullseye | |||
* | * 21:21 brennen@deploy1002: brennen and nray: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet | ||
* | * 21:20 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1078.eqiad.wmnet with reason: host reimage | ||
* 21:19 brennen@deploy1002: Started scap: Backport for [[gerrit:886118{{!}}Enable client preferences everywhere (T327979)]] | |||
* | * 21:18 brennen@deploy1002: Finished scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] (duration: 12m 02s) | ||
* | * 21:07 brennen@deploy1002: brennen and dreamyjazz: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet | ||
* 21:06 brennen@deploy1002: Started scap: Backport for [[gerrit:885359{{!}}Disable write old for CheckUserLog reason everywhere (T233004)]] | |||
* | * 20:59 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye | ||
* 20:59 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1078.eqiad.wmnet with OS bullseye | |||
* | * 20:52 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage | ||
* 20:49 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1077.eqiad.wmnet with reason: host reimage | |||
* | * 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1078.eqiad.wmnet with OS bullseye | ||
* | * 20:28 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp1077.eqiad.wmnet with OS bullseye | ||
* | * 20:23 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include bullseye-wikimedia /home/rzl/httpbb/bullseye/httpbb_0.0.3-1+deb11u1_amd64.changes # [[phab:T328280|T328280]] | ||
* | * 20:21 rzl: rzl@apt1001:~$ sudo -i reprepro -C main include buster-wikimedia /home/rzl/httpbb/buster/httpbb_0.0.3-1_amd64.changes # [[phab:T328280|T328280]] | ||
* | * 20:11 zabe@deploy1002: Finished scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] (duration: 09m 39s) | ||
* | * 20:03 zabe@deploy1002: zabe: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet | ||
* | * 20:02 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host elastic2037.codfw.wmnet | ||
* | * 20:01 zabe@deploy1002: Started scap: Backport for [[gerrit:886135{{!}}Stop writing to cuc_user and cuc_user_text everywhere (T233004)]] | ||
* | * 19:55 bking@cumin1001: START - Cookbook sre.hosts.reboot-single for host elastic2037.codfw.wmnet | ||
* | * 19:54 ryankemper: [[phab:T328674|T328674]] [Elastic] With puppet disabled on elastic* fleet, `ryankemper@elastic2037:~$ sudo run-puppet-agent --force` to verify changes in https://gerrit.wikimedia.org/r/886055 | ||
* 19:30 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.21 refs [[phab:T325584|T325584]] | |||
* | * 19:28 zabe@deploy1002: say aborted: (duration: 00m 03s) | ||
* | * 18:42 zabe@deploy1002: Finished scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] (duration: 08m 19s) | ||
* 18:36 zabe@deploy1002: zabe: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet | |||
* | * 18:34 zabe@deploy1002: Started scap: Backport for [[gerrit:886127{{!}}Stop writing to cuc_comment in group1 wikis (T233004)]] | ||
* | * 18:08 aokoth@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0 | ||
* | * 18:08 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply | ||
* | * 18:08 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply | ||
* | * 18:08 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2043.codfw.wmnet with OS bullseye | ||
* | * 18:07 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply | ||
* 18:06 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply | |||
* | * 18:05 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply | ||
* | * 18:05 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply | ||
* | * 18:03 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1037.eqiad.wmnet with OS bullseye | ||
* | * 17:52 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage | ||
* | * 17:49 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2043.codfw.wmnet with reason: host reimage | ||
* | * 17:47 jiji@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage | ||
* | * 17:45 jiji@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on mc1037.eqiad.wmnet with reason: host reimage | ||
* | * 17:33 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc2043.codfw.wmnet with OS bullseye | ||
* | * 17:32 jiji@cumin1001: START - Cookbook sre.hosts.reimage for host mc1037.eqiad.wmnet with OS bullseye | ||
* | * 17:29 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Production (gitlab1004) to 15.7.6-ce.0 | ||
* | * 17:12 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: sync | ||
* | * 17:12 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: sync | ||
* | * 16:53 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply | ||
* | * 16:52 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply | ||
* 16:51 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply | |||
* | * 16:50 dancy@deploy1002: Installation of scap version "4.34.0" completed for 561 hosts | ||
* 16:50 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply | |||
* | * 16:50 dancy@deploy1002: Installing scap version "4.34.0" for 561 hosts | ||
* | * 16:50 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 16:49 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 16:48 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 16:48 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 16:47 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: sync | ||
* | * 16:46 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: sync | ||
* | * 16:25 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2007.codfw.wmnet | ||
* | * 16:18 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply | ||
* | * 16:17 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply | ||
* | * 16:17 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2007.codfw.wmnet | ||
* | * 16:17 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply | ||
* | * 16:16 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply | ||
* | * 16:16 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 16:15 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 16:10 volans: uploaded python3-wmflib_1.2.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia | ||
* | * 16:10 dzahn@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0 | ||
* | * 15:40 jnuche@deploy1002: Finished deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided) (duration: 07m 01s) | ||
* | * 15:38 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:37 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:35 aokoth@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:35 aokoth@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release | ||
* | * 15:34 dzahn@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab Replica gitlab2002 to 15.7.6-ce.0 | ||
* | * 15:33 jnuche@deploy1002: Started deploy [releng/jenkins-deploy@e38efa6] (releasing): (no justification provided) | ||
* | * 15:24 jmm@cumin2002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti3004 | ||
* 15:17 jmm@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti3004 | |||
* | * 15:06 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2006.codfw.wmnet | ||
* | * 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | ||
* | * 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002" | ||
* 15:02 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast3004 was renamed as ganeti4004 - jmm@cumin2002" | |||
* | * 15:00 vgutierrez: rolling restart of varnish in cache::text - [[phab:T315676|T315676]] | ||
* | * 14:59 jmm@cumin2002: START - Cookbook sre.dns.netbox | ||
* | * 14:59 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2006.codfw.wmnet | ||
* | * 14:55 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* 14:45 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | |||
* | * 14:39 cgoubert@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | ||
* | * 14:31 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2005.codfw.wmnet | ||
* | * 14:29 cgoubert@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 14:25 moritzm: installing containerd security updates on codfw k8s nodes | ||
* | * 14:24 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2005.codfw.wmnet | ||
* | * 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=ats-be | ||
* | * 13:34 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet,service=cdn | ||
* | * 13:10 kharlan:: Deployed security patch for [[phab:T328643|T328643]] | ||
* | * 13:09 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1076.eqiad.wmnet with OS bullseye | ||
* | * 13:04 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . | ||
* | * 13:03 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . | ||
* | * 13:03 kharlan:: Deployed security patch for [[phab:T328643|T328643]] | ||
* | * 13:02 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . | ||
* | * 13:01 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2004.codfw.wmnet | ||
* 13:00 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . | |||
* | * 12:55 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2004.codfw.wmnet | ||
* | * 12:47 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage | ||
* | * 12:47 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . | ||
* | * 12:46 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . | ||
* | * 12:44 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1076.eqiad.wmnet with reason: host reimage | ||
* | * 12:42 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | ||
* | * 12:42 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | ||
* | * 12:39 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . | ||
* | * 12:39 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . | ||
* | * 12:29 btullis@deploy1002: Finished deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade (duration: 00m 42s) | ||
* | * 12:29 claime: Work ongoing on m2 and m3 | ||
* | * 12:29 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2003.codfw.wmnet | ||
* | * 12:29 btullis@deploy1002: Started deploy [analytics/superset/deploy@5175ad7]: Production deployment for numpy downgrade | ||
* | * 12:23 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1076.eqiad.wmnet with OS bullseye | ||
* | * 12:22 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2003.codfw.wmnet | ||
* | * 12:08 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . | ||
* | * 12:08 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . | ||
* | * 11:46 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . | ||
* 11:42 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply | |||
* | * 11:42 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply | ||
* 11:41 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply | |||
* | * 11:41 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply | ||
* | * 11:40 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply | ||
* | * 11:39 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply | ||
* | * 11:38 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply | ||
* | * 11:37 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply | ||
* | * 11:37 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-4.out # [[phab:T328634|T328634]] – made some progress then errored out again | ||
* | * 11:32 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-3.out # [[phab:T328634|T328634]] – seemed to finish the first 20 pages and then go into an infinite loop, I Ctrl+Ced it | ||
* | * 11:28 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix --add-prefix=[[phab:T328634|T328634]]/ {{!}} tee [[phab:T328634|T328634]]-namespaceDupes-2.out # [[phab:T328634|T328634]] – another error but made more progress | ||
* | * 11:23 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php shnwikibooks --fix {{!}} tee [[phab:T328634|T328634]]-namespaceDupes.out # [[phab:T328634|T328634]] – failed quickly, details in task | ||
* 11:22 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop: sync | |||
* | * 11:22 elukey@deploy1002: helmfile [staging] START helmfile.d/services/changeprop: sync | ||
* 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply | |||
* | * 11:02 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply | ||
* | * 10:27 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2002.codfw.wmnet | ||
* | * 10:19 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2002.codfw.wmnet | ||
* | * 10:17 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . | ||
* 10: | * 10:11 moritzm: restarting FPM on mw canaries to pick up tiff security updates | ||
* 10: | * 10:04 moritzm: installing tiff security updates | ||
* 10: | * 09:59 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2001.codfw.wmnet | ||
* 10: | * 09:55 elukey@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync | ||
* | * 09:54 elukey@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: sync | ||
* | * 09:51 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host aqs2001.codfw.wmnet | ||
* | * 09:40 elukey@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync | ||
* | * 09:40 elukey@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: sync | ||
* 09:19 ayounsi@cumin1001: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 398143 | |||
* | * 09:19 ayounsi@cumin1001: START - Cookbook sre.network.peering with action 'configure' for AS: 398143 | ||
* | * 09:16 jelto@cumin1001: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6 | ||
* | * 09:13 apergos: UTC morning backport and config training window done | ||
* | * 09:13 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-main: sync | ||
* | * 09:12 elukey@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-main: sync | ||
* | * 09:11 elukey: roll restart of eventgate-main pods in wikikube eqiad/codfw to pick up new stream configs - [[phab:T328576|T328576]] | ||
* | * 08:57 ariel@deploy1002: Finished scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] (duration: 10m 56s) | ||
* | * 08:48 ariel@deploy1002: ariel and aishik: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet | ||
* 08:46 ariel@deploy1002: Started scap: Backport for [[gerrit:885927{{!}}Enable wgMinervaEnableSiteNotice for bnwiktionary (T328630)]] | |||
* | * 08:39 jelto@cumin1001: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica gitlab1004 to 15.7.6 | ||
* 08:37 tgr@deploy1002: Finished scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] (duration: 14m 26s) | |||
* | * 08:27 tgr@deploy1002: tgr: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | ||
* 08:23 tgr@deploy1002: Started scap: Backport for [[gerrit:885928{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]], [[gerrit:885929{{!}}campaigns: Donor landing page translations for sv, it, ja, fr, nl (T321370)]] | |||
* | * 06:17 kart_: Updated cxserver to 2023-02-02-004918-production ([[phab:T129470|T129470]], [[phab:T172035|T172035]], [[phab:T327842|T327842]]) | ||
* 06:16 kartik@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply | |||
* | * 06:15 kartik@deploy1002: helmfile [eqiad] START helmfile.d/services/cxserver: apply | ||
* | * 06:13 kartik@deploy1002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply | ||
* | * 06:12 kartik@deploy1002: helmfile [codfw] START helmfile.d/services/cxserver: apply | ||
* 09 | * 06:09 kartik@deploy1002: helmfile [staging] DONE helmfile.d/services/cxserver: apply | ||
* | * 06:09 kartik@deploy1002: helmfile [staging] START helmfile.d/services/cxserver: apply | ||
* | * 04:00 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5024.eqsin.wmnet | ||
* | * 03:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5024.eqsin.wmnet with OS bullseye | ||
* | * 03:21 ejegg: payments-wiki upgraded from {{Gerrit|f20a2208}} to {{Gerrit|53d1a58d}} | ||
* | * 02:49 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage | ||
* 02:46 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5024.eqsin.wmnet with reason: host reimage | |||
* | * 02:14 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye | ||
* 02:14 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5024.eqsin.wmnet with OS bullseye | |||
* | * 01:56 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5024.eqsin.wmnet with OS bullseye | ||
* | * 01:55 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5023.eqsin.wmnet | ||
* | * 01:55 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5023.eqsin.wmnet with OS bullseye | ||
* | * 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be | ||
* 01:50 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=cdn | |||
* | * 01:49 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1075.eqiad.wmnet with OS bullseye | ||
* | * 01:27 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage | ||
* | * 01:24 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1075.eqiad.wmnet with reason: host reimage | ||
* 01:21 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage | |||
* 01:18 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5023.eqsin.wmnet with reason: host reimage | |||
* 01:07 sukhe@cumin2002: START - Cookbook sre.hosts.reimage for host cp1075.eqiad.wmnet with OS bullseye | |||
* 00:44 brett@cumin2002: START - Cookbook sre.hosts.reimage for host cp5023.eqsin.wmnet with OS bullseye | |||
* 00:06 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5022.eqsin.wmnet | |||
* 00:04 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS bullseye | |||
== 2023-02-01 == | |||
* 23:45 zabe@deploy1002: Finished scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] (duration: 08m 07s) | |||
* 23:39 zabe@deploy1002: zabe: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet | |||
* 23:37 zabe@deploy1002: Started scap: Backport for [[gerrit:885908{{!}}Stop writing to cuc_user and cuc_user_text in group1 wikis (T233004)]] | |||
* 23:31 rzl@cumin2002: dbctl commit (dc=all): 'Depool db2181', diff saved to https://phabricator.wikimedia.org/P43574 and previous config saved to /var/cache/conftool/dbconfig/20230201-233140-rzl.json | |||
* 23:31 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage | |||
* 23:27 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage | |||
* 23:19 dzahn@cumin1001: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia. | |||
==Archives == | |||