You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Difference between revisions of "Server Admin Log"

From Wikitech
Jump to navigation Jump to search
imported>Stashbot
(ejegg: updated payments-wiki from 5432f9c3a4 to 4ebbdb247d)
imported>Stashbot
(XioNoX: enable netflow sampling on cr1-codfw)
Line 1: Line 1:
 +
== 2019-09-12 ==
 +
* 23:35 XioNoX: enable netflow sampling on cr1-codfw
 +
* 23:21 urandom: decommissioning Cassandra, restbase2009-b -- [[phab:T224553|T224553]]
 +
* 23:19 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: [[phab:T223602|T223602]] Read config from JSON, not serialised PHP on testwiki (duration: 01m 03s)
 +
* 23:18 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: [[phab:T223602|T223602]] Add ability to read config from JSON, not serialised PHP (duration: 01m 04s)
 +
* 23:10 eileen: process-control config revision is {{Gerrit|1da8391a9a}}
 +
* 22:53 ayounsi@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
 +
* 22:48 ayounsi@cumin2001: START - Cookbook sre.ganeti.makevm
 +
* 22:43 ayounsi@cumin2001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
 +
* 22:43 ayounsi@cumin2001: START - Cookbook sre.ganeti.makevm
 +
* 22:20 XenoRyet: payments-wiki updated from {{Gerrit|4ebbdb247d}} to {{Gerrit|1f556670cf}}
 +
* 22:14 XioNoX: remove extra prepend in AMS-IX
 +
* 21:18 hashar@deploy1001: Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Hardcode posix signal and log coredump - [[phab:T232613|T232613]] (duration: 01m 04s)
 +
* 21:17 mbsantos@deploy1001: Finished deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0 (duration: 03m 18s)
 +
* 21:14 mbsantos@deploy1001: Started deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0
 +
* 21:13 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0 (duration: 03m 52s)
 +
* 21:09 mbsantos@deploy1001: Started deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0
 +
* 21:00 urandom: decommissioning Cassandra, restbase2009 -- [[phab:T224553|T224553]]
 +
* 20:33 krinkle@deploy1001: Synchronized wmf-config/: {{Gerrit|d495d5e24949}} (duration: 01m 03s)
 +
* 20:28 krinkle@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: {{Gerrit|d495d5e24949}} (duration: 01m 04s)
 +
* 20:27 eileen: civicrm revision changed from {{Gerrit|4075e396d5}} to {{Gerrit|f00c6482bf}}, config revision is {{Gerrit|635f198b92}}
 +
* 20:05 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: beta-only (duration: 01m 02s)
 +
* 20:03 krinkle@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: beta-only (duration: 01m 04s)
 +
* 20:02 moritzm: installing firmware-nonfree update from Buster 10.1 point release
 +
* 19:51 moritzm: installing systemd bugfix update from Buster 10.1 point release
 +
* 19:44 moritzm: installing 4.19.67 kernel from 10.1 point release on Buster systems
 +
* 19:34 urandom: bootstrapping Cassandra, restbase1018-c -- [[phab:T224553|T224553]]
 +
* 18:59 hashar@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable coredump on some mysterious php7.2 failure - [[phab:T232613|T232613]] (duration: 01m 04s)
 +
* 18:32 moritzm: installing gdb updates from buster 10.1 point release
 +
* 18:28 bblack: lvs1016: restart pybal to revert test
 +
* 18:21 bblack: lvs1016: restart pybal to test dual bgp peering
 +
* 18:04 bblack: lvs1015: restart pybal to return BGP session to cr2 - [[phab:T226424|T226424]]
 +
* 18:03 bblack: lvs1014: restart pybal to return BGP session to cr2 - [[phab:T226424|T226424]]
 +
* 17:58 XioNoX: revert VRRP priority change cr2-eqiad - [[phab:T226424|T226424]]
 +
* 17:54 XioNoX: revert OSPF priority change on cr2-eqiad - [[phab:T226424|T226424]]
 +
* 17:53 XioNoX: re-enabled external BGP on cr2-eqiad - [[phab:T226424|T226424]]
 +
* 17:46 urandom: bootstrapping Cassandra, restbase1018-b -- [[phab:T224553|T224553]]
 +
* 17:43 XioNoX: reboot cr2-eqiad - [[phab:T226424|T226424]]
 +
* 17:40 XioNoX: failover cr2-eqiad master RE from RE1 to RE0 - [[phab:T226424|T226424]]
 +
* 17:31 jforrester@deploy1001: Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: [[phab:T232613|T232613]] Add ability to core dump on empty string array key that should exist (wmf.22 only, flagged off) (duration: 01m 03s)
 +
* 17:31 XioNoX: power off re0.cr2-eqiad - [[phab:T226424|T226424]]
 +
* 17:25 XioNoX: failover cr2-eqiad master RE from RE0 to RE1 - [[phab:T226424|T226424]]
 +
* 17:19 halfak@deploy1001: Finished deploy [ores/deploy@7d45b80]: [[phab:T232660|T232660]] (duration: 13m 41s)
 +
* 17:05 halfak@deploy1001: Started deploy [ores/deploy@7d45b80]: [[phab:T232660|T232660]]
 +
* 17:04 XioNoX: power off re1.cr2-eqiad - [[phab:T226424|T226424]]
 +
* 17:02 moritzm: installing unzip security updates on buster
 +
* 17:00 XioNoX: +1000 metric to all transport to/from cr2-eqiad - [[phab:T226424|T226424]]
 +
* 16:57 moritzm: installing libxslt security updates on buster
 +
* 16:49 XioNoX: Deactivate IX/transit/private-peer v4/v6 BGP on cr2-eqiad - [[phab:T226424|T226424]]
 +
* 16:47 moritzm: installing NSS security updates on buster
 +
* 16:42 XioNoX: er, switch VRRP master to cr1-eqiad - [[phab:T226424|T226424]]
 +
* 16:42 XioNoX: switch VRRP master to cr2-eqiad - [[phab:T226424|T226424]]
 +
* 16:36 bblack: lvs1013: restart pybal to move bgp session to cr1 - [[phab:T226424|T226424]]
 +
* 16:36 bblack: lvs1014: restart pybal to move bgp session to cr1 - [[phab:T226424|T226424]]
 +
* 16:35 bblack: lvs1015: restart pybal to move bgp session to cr1 - [[phab:T226424|T226424]]
 +
* 16:34 bblack: lvs1016: restart pybal to move bgp session to cr1 - [[phab:T226424|T226424]]
 +
* 16:19 XioNoX: rollback force VRRP backup on cr1-eqiad - [[phab:T226424|T226424]]
 +
* 16:16 XioNoX: activate CF tunnel on cr1-eqiad - [[phab:T226424|T226424]]
 +
* 16:16 XioNoX: activate transit4/6 on cr1-eqiad - [[phab:T226424|T226424]]
 +
* 16:09 urandom: bootstrapping Cassandra, restbase1018-a -- [[phab:T224553|T224553]]
 +
* 16:04 XioNoX: reboot cr1-eqiad - [[phab:T226424|T226424]]
 +
* 16:01 XioNoX: force offline/online of FPC3 on cr1-eqiad
 +
* 15:45 XioNoX: failover master RE from RE1 to RE0 on cr1-eqiad - [[phab:T226424|T226424]]
 +
* 15:39 XioNoX: deactivate transit4/6 on cr1-eqiad - [[phab:T226424|T226424]]
 +
* 15:31 XioNoX: shutdown re0.cr1-eqiad - [[phab:T226424|T226424]]
 +
* 15:23 XioNoX: failover master RE from RE0 to RE1 on cr1-eqiad - [[phab:T226424|T226424]]
 +
* 15:13 XioNoX: shutdown re1.cr1-eqiad - [[phab:T226424|T226424]]
 +
* 15:05 XioNoX: disable primary tunnel to CF in eqiad (for real this time, I did see an uptake of traffic on backup link before the rollback)
 +
* 15:03 XioNoX: rolled back disable primary tunnel to CF in eqiad
 +
* 15:02 XioNoX: disable primary tunnel to CF in eqiad
 +
* 14:53 bblack: restart pybal on lvs1013 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - [[phab:T226424|T226424]]
 +
* 14:50 bblack: restart pybal on lvs1016 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - [[phab:T226424|T226424]]
 +
* 14:45 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 14:41 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
 +
* 14:39 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 14:37 akosiaris@: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 14:29 XioNoX: ensure cr1-eqiad is vrrp backup for all groups - [[phab:T226424|T226424]]
 +
* 13:22 akosiaris@: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
 +
* 13:03 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
 +
* 13:01 jmm@cumin1001: START - Cookbook sre.hosts.downtime
 +
* 12:57 effie: restarting hhvm on mw1233 and repooling
 +
* 12:56 effie: depool mw12333
 +
* 12:38 moritzm: reimaging restbase1018 to stretch
 +
* 12:03 Amir1: EU SWAT is done
 +
* 12:03 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:536167{{!}}Set item terms on write both up to Q20mio (T225055)]] (duration: 01m 31s)
 +
* 11:11 akosiaris@: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 11:11 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 11:09 akosiaris@: helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 11:09 akosiaris@: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 11:00 akosiaris@: helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' .
 +
* 09:42 jynus: compressing tables on labsdb1012 [[phab:T232446|T232446]]
 +
* 08:22 vgutierrez: upgrading to acme-chief 0.21 on acmechief-test instances - [[phab:T219765|T219765]]
 +
* 08:17 vgutierrez: restarting pybal on lvs1015 and lvs2003 - [[phab:T176875|T176875]]
 +
* 08:13 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=wdqs,service=wdqs-heavy-queries
 +
* 08:11 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=puppetmaster1001.eqiad.wmnet,service=wdqs-heavy-queries
 +
* 08:07 vgutierrez: restarting pybal on lvs2006 - [[phab:T176875|T176875]]
 +
* 08:02 vgutierrez: restarting pybal on lvs1016 - [[phab:T176875|T176875]]
 +
* 07:45 vgutierrez: uploaded acme-chief 0.21 to apt.wikimedia.org (buster) - [[phab:T219765|T219765]]
 +
* 06:51 vgutierrez: restarting ATS-TLS on cp4021 and cp2002 to get the new SSL session cache size - [[phab:T232298|T232298]]
 +
* 06:00 marostegui: Stop MySQL on db1073 for decommission [[phab:T231892|T231892]]
 +
* 05:59 marostegui: Remove db1073 from tendril and zarcillo [[phab:T231892|T231892]]
 +
* 05:26 _joe_: restarting strongswan on all eqiad caches that need it
 +
* 05:23 _joe_: restarting strongswan on cp1077
 +
* 03:37 eileen: civicrm revision changed from {{Gerrit|32cd5e4953}} to {{Gerrit|4075e396d5}}, config revision is {{Gerrit|3e22a80bc8}}
 +
* 02:13 eileen: civicrm revision changed from {{Gerrit|53aeba6318}} to {{Gerrit|32cd5e4953}}, config revision is {{Gerrit|3e22a80bc8}}
 +
* 02:03 XioNoX: repooling ulsfo
 +
 
== 2019-09-11 ==
 
== 2019-09-11 ==
 
* 23:50 ejegg: updated payments-wiki from {{Gerrit|5432f9c3a4}} to {{Gerrit|4ebbdb247d}}
 
* 23:50 ejegg: updated payments-wiki from {{Gerrit|5432f9c3a4}} to {{Gerrit|4ebbdb247d}}

Revision as of 23:35, 12 September 2019

2019-09-12

  • 23:35 XioNoX: enable netflow sampling on cr1-codfw
  • 23:21 urandom: decommissioning Cassandra, restbase2009-b -- T224553
  • 23:19 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T223602 Read config from JSON, not serialised PHP on testwiki (duration: 01m 03s)
  • 23:18 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: T223602 Add ability to read config from JSON, not serialised PHP (duration: 01m 04s)
  • 23:10 eileen: process-control config revision is 1da8391a9a
  • 22:53 ayounsi@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • 22:48 ayounsi@cumin2001: START - Cookbook sre.ganeti.makevm
  • 22:43 ayounsi@cumin2001: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99)
  • 22:43 ayounsi@cumin2001: START - Cookbook sre.ganeti.makevm
  • 22:20 XenoRyet: payments-wiki updated from 4ebbdb247d to 1f556670cf
  • 22:14 XioNoX: remove extra prepend in AMS-IX
  • 21:18 hashar@deploy1001: Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Hardcode posix signal and log coredump - T232613 (duration: 01m 04s)
  • 21:17 mbsantos@deploy1001: Finished deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0 (duration: 03m 18s)
  • 21:14 mbsantos@deploy1001: Started deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0
  • 21:13 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0 (duration: 03m 52s)
  • 21:09 mbsantos@deploy1001: Started deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0
  • 21:00 urandom: decommissioning Cassandra, restbase2009 -- T224553
  • 20:33 krinkle@deploy1001: Synchronized wmf-config/: d495d5e24949 (duration: 01m 03s)
  • 20:28 krinkle@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: d495d5e24949 (duration: 01m 04s)
  • 20:27 eileen: civicrm revision changed from 4075e396d5 to f00c6482bf, config revision is 635f198b92
  • 20:05 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: beta-only (duration: 01m 02s)
  • 20:03 krinkle@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: beta-only (duration: 01m 04s)
  • 20:02 moritzm: installing firmware-nonfree update from Buster 10.1 point release
  • 19:51 moritzm: installing systemd bugfix update from Buster 10.1 point release
  • 19:44 moritzm: installing 4.19.67 kernel from 10.1 point release on Buster systems
  • 19:34 urandom: bootstrapping Cassandra, restbase1018-c -- T224553
  • 18:59 hashar@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable coredump on some mysterious php7.2 failure - T232613 (duration: 01m 04s)
  • 18:32 moritzm: installing gdb updates from buster 10.1 point release
  • 18:28 bblack: lvs1016: restart pybal to revert test
  • 18:21 bblack: lvs1016: restart pybal to test dual bgp peering
  • 18:04 bblack: lvs1015: restart pybal to return BGP session to cr2 - T226424
  • 18:03 bblack: lvs1014: restart pybal to return BGP session to cr2 - T226424
  • 17:58 XioNoX: revert VRRP priority change cr2-eqiad - T226424
  • 17:54 XioNoX: revert OSPF priority change on cr2-eqiad - T226424
  • 17:53 XioNoX: re-enabled external BGP on cr2-eqiad - T226424
  • 17:46 urandom: bootstrapping Cassandra, restbase1018-b -- T224553
  • 17:43 XioNoX: reboot cr2-eqiad - T226424
  • 17:40 XioNoX: failover cr2-eqiad master RE from RE1 to RE0 - T226424
  • 17:31 jforrester@deploy1001: Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: T232613 Add ability to core dump on empty string array key that should exist (wmf.22 only, flagged off) (duration: 01m 03s)
  • 17:31 XioNoX: power off re0.cr2-eqiad - T226424
  • 17:25 XioNoX: failover cr2-eqiad master RE from RE0 to RE1 - T226424
  • 17:19 halfak@deploy1001: Finished deploy [ores/deploy@7d45b80]: T232660 (duration: 13m 41s)
  • 17:05 halfak@deploy1001: Started deploy [ores/deploy@7d45b80]: T232660
  • 17:04 XioNoX: power off re1.cr2-eqiad - T226424
  • 17:02 moritzm: installing unzip security updates on buster
  • 17:00 XioNoX: +1000 metric to all transport to/from cr2-eqiad - T226424
  • 16:57 moritzm: installing libxslt security updates on buster
  • 16:49 XioNoX: Deactivate IX/transit/private-peer v4/v6 BGP on cr2-eqiad - T226424
  • 16:47 moritzm: installing NSS security updates on buster
  • 16:42 XioNoX: er, switch VRRP master to cr1-eqiad - T226424
  • 16:42 XioNoX: switch VRRP master to cr2-eqiad - T226424
  • 16:36 bblack: lvs1013: restart pybal to move bgp session to cr1 - T226424
  • 16:36 bblack: lvs1014: restart pybal to move bgp session to cr1 - T226424
  • 16:35 bblack: lvs1015: restart pybal to move bgp session to cr1 - T226424
  • 16:34 bblack: lvs1016: restart pybal to move bgp session to cr1 - T226424
  • 16:19 XioNoX: rollback force VRRP backup on cr1-eqiad - T226424
  • 16:16 XioNoX: activate CF tunnel on cr1-eqiad - T226424
  • 16:16 XioNoX: activate transit4/6 on cr1-eqiad - T226424
  • 16:09 urandom: bootstrapping Cassandra, restbase1018-a -- T224553
  • 16:04 XioNoX: reboot cr1-eqiad - T226424
  • 16:01 XioNoX: force offline/online of FPC3 on cr1-eqiad
  • 15:45 XioNoX: failover master RE from RE1 to RE0 on cr1-eqiad - T226424
  • 15:39 XioNoX: deactivate transit4/6 on cr1-eqiad - T226424
  • 15:31 XioNoX: shutdown re0.cr1-eqiad - T226424
  • 15:23 XioNoX: failover master RE from RE0 to RE1 on cr1-eqiad - T226424
  • 15:13 XioNoX: shutdown re1.cr1-eqiad - T226424
  • 15:05 XioNoX: disable primary tunnel to CF in eqiad (for real this time, I did see an uptake of traffic on backup link before the rollback)
  • 15:03 XioNoX: rolled back disable primary tunnel to CF in eqiad
  • 15:02 XioNoX: disable primary tunnel to CF in eqiad
  • 14:53 bblack: restart pybal on lvs1013 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - T226424
  • 14:50 bblack: restart pybal on lvs1016 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - T226424
  • 14:45 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 14:41 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
  • 14:39 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 14:37 akosiaris@: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 14:29 XioNoX: ensure cr1-eqiad is vrrp backup for all groups - T226424
  • 13:22 akosiaris@: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
  • 13:03 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 13:01 jmm@cumin1001: START - Cookbook sre.hosts.downtime
  • 12:57 effie: restarting hhvm on mw1233 and repooling
  • 12:56 effie: depool mw12333
  • 12:38 moritzm: reimaging restbase1018 to stretch
  • 12:03 Amir1: EU SWAT is done
  • 12:03 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set item terms on write both up to Q20mio (T225055) (duration: 01m 31s)
  • 11:11 akosiaris@: helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 11:11 akosiaris@: helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 11:09 akosiaris@: helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 11:09 akosiaris@: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 11:00 akosiaris@: helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 09:42 jynus: compressing tables on labsdb1012 T232446
  • 08:22 vgutierrez: upgrading to acme-chief 0.21 on acmechief-test instances - T219765
  • 08:17 vgutierrez: restarting pybal on lvs1015 and lvs2003 - T176875
  • 08:13 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=wdqs,service=wdqs-heavy-queries
  • 08:11 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=puppetmaster1001.eqiad.wmnet,service=wdqs-heavy-queries
  • 08:07 vgutierrez: restarting pybal on lvs2006 - T176875
  • 08:02 vgutierrez: restarting pybal on lvs1016 - T176875
  • 07:45 vgutierrez: uploaded acme-chief 0.21 to apt.wikimedia.org (buster) - T219765
  • 06:51 vgutierrez: restarting ATS-TLS on cp4021 and cp2002 to get the new SSL session cache size - T232298
  • 06:00 marostegui: Stop MySQL on db1073 for decommission T231892
  • 05:59 marostegui: Remove db1073 from tendril and zarcillo T231892
  • 05:26 _joe_: restarting strongswan on all eqiad caches that need it
  • 05:23 _joe_: restarting strongswan on cp1077
  • 03:37 eileen: civicrm revision changed from 32cd5e4953 to 4075e396d5, config revision is 3e22a80bc8
  • 02:13 eileen: civicrm revision changed from 53aeba6318 to 32cd5e4953, config revision is 3e22a80bc8
  • 02:03 XioNoX: repooling ulsfo

2019-09-11

  • 23:50 ejegg: updated payments-wiki from 5432f9c3a4 to 4ebbdb247d
  • 23:20 XioNoX: `set protocols bgp group Netflow cluster 208.80.154.197` on cr2-eqiad
  • 22:43 XioNoX: `set protocols bgp group Netflow cluster 208.80.154.196` on cr1-eqiad
  • 22:36 XioNoX: add BGP session between cr2-eqord and netflow1001
  • 22:30 urandom: decommissioning Cassandra, restbase1018-c -- T224553
  • 20:57 urandom: bootstrapping Cassandra, restbase-dev1005-b -- T224554
  • 20:21 ottomata: stopped and removed eventlogging-service-eventbus - T232122
  • 20:12 ppchelko@deploy1001: Finished deploy [changeprop/deploy@522177f]: Clean up old event style support (duration: 01m 39s)
  • 20:11 ppchelko@deploy1001: Started deploy [changeprop/deploy@522177f]: Clean up old event style support
  • 20:07 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@2c9e409]: Clean up old event style support T230049 (duration: 00m 53s)
  • 20:06 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@2c9e409]: Clean up old event style support T230049
  • 18:43 urandom: decommissioning Cassandra, restbase1018-b -- T224553
  • 18:42 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211124 ed8dd7aad9e5 (duration: 01m 04s)
  • 18:42 nuria@deploy1001: Finished deploy [analytics/refinery@fa994c7]: v0.0.99 of refinery, again, try II. last time shas commited by jenkins were incorrect (duration: 08m 39s)
  • 18:40 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: no-op ed8dd7aad9e5 (duration: 01m 06s)
  • 18:37 krinkle@deploy1001: Synchronized tests/: no-op ed8dd7aad9e5 (duration: 01m 05s)
  • 18:33 nuria@deploy1001: Started deploy [analytics/refinery@fa994c7]: v0.0.99 of refinery, again, try II. last time shas commited by jenkins were incorrect
  • 18:16 krinkle@deploy1001: Synchronized wmf-config/logging.php: d6865e3365e8 - T211124 (duration: 01m 04s)
  • 18:16 nuria@deploy1001: Finished deploy [analytics/refinery@f4c60a4]: v0.0.99 of refinery (duration: 01m 21s)
  • 18:15 nuria@deploy1001: Started deploy [analytics/refinery@f4c60a4]: v0.0.99 of refinery
  • 18:02 krinkle@deploy1001: Synchronized php-1.34.0-wmf.22/extensions/WikimediaMaintenance/blameStartupRegistry.php: (no justification provided) (duration: 01m 05s)
  • 17:57 XioNoX: upgrade librenms to 1.55
  • 17:43 ayounsi@deploy1001: Finished deploy [librenms/librenms@2a06e98]: Upgrade LibreNMS to 1.55 - T232599 (duration: 00m 09s)
  • 17:42 ayounsi@deploy1001: Started deploy [librenms/librenms@2a06e98]: Upgrade LibreNMS to 1.55 - T232599
  • 17:32 bblack: enable GRE MTU mitigation on eqsin caches (cp5xxx) - T232602
  • 17:27 bblack: restbase2009 - re-pool - T227408
  • 17:07 bblack: restbase2009 - shutdown for hardware work - T227408
  • 17:05 bblack: restbase2009 - depool for hardware work - T227408
  • 16:57 urbanecm@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/modules/homepage/ext.growthExperiments.StartModule.less: SWAT: c0fd061: Homepage: Fix start module layout bugs (T230629, T232549, T225668) (duration: 01m 02s)
  • 16:54 bblack: manually removed decommed eventbus LVS IP on kafka100[23]
  • 16:54 bblack: manually removed decommed eventbus LVS IP on kafka-main1001
  • 16:50 bblack: manually removed decommed eventbus LVS IP on kafka-main200[23]
  • 16:49 bblack: manually removed decommed eventbus LVS IP on kafka-main2001
  • 16:42 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 6007fbc: [rowiki] Allow sysops to remove patrollers (T231099) (duration: 01m 03s)
  • 16:39 urandom: decommissioning Cassandra, restbase1018-a -- T224553
  • 16:38 Urbanecm: Run mwscript emptyUserGroup.php --wiki=fawiki OTRS-member (T232554)
  • 16:36 bblack: ran conftool-merge on puppetmaster1001 (manually from sudo -i, to fixup missing updates)
  • 16:35 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 76991f2: Remove OTRS-member usergroup from fawiki (T232554) (duration: 01m 05s)
  • 16:32 Urbanecm: mwscript importImages.php --wiki=commonswiki --user=Abbe98 --comment-ext=txt /home/urbanecm/T232346
  • 16:31 urbanecm@deploy1001: Synchronized php-1.34.0-wmf.22/extensions/GrowthExperiments/modules/homepage/ext.growthExperiments.StartModule.less: SWAT: c45d6d0: Homepage: Fix start module layout bugs (T230629, T232549, T225668) (duration: 01m 03s)
  • 16:28 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 565fafa: Set noindex for user and user_talk on zhwiki (T231982) (duration: 01m 05s)
  • 16:24 urandom: bootstrapping Cassandra, restbase-dev1005-a -- T224554
  • 16:16 bblack@cumin1001: conftool action : set/pooled=no; selector: cluster=eventbus
  • 16:10 urbanecm@deploy1001: Synchronized wmf-config/throttle.php: SWAT: 510aa6b: Add new whitelist rule for Université de Lorraine course (T232596) (duration: 01m 04s)
  • 16:07 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: eceaccf: Add autopatrolled user group to az.wikibooks (T231493) (duration: 01m 06s)
  • 15:52 bblack: lvs1015 - remove eventbus.svc.eqiad.wmnet service, restart pybal, etc
  • 15:51 bblack: lvs2003 - remove eventbus.svc.codfw.wmnet service, restart pybal, etc
  • 15:49 bblack: lvs1016 - remove eventbus.svc.eqiad.wmnet service, restart pybal, etc
  • 15:48 bblack: lvs2006 - remove eventbus.svc.codfw.wmnet service, restart pybal, etc
  • 15:03 bblack: downtimed dns-discovery confd health checks for eventbus - T232122
  • 13:13 hashar@deploy1001: Synchronized php: group1 wikis to 1.34.0-wmf.22 (duration: 01m 02s)
  • 13:12 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.22
  • 12:48 moritzm: upgrade labpuppetmaster* to use facter 3 / puppet 5
  • 12:40 moritzm: removing now obsolete puppet/puppetdb packages from labpuppetmaster* T171188
  • 12:40 moritzm: removing now puppet/puppetdb packages from labpuppetmaster* T171188
  • 11:59 hashar: Restarting Gerrit due to deadlock in the account cache # T224448
  • 11:57 bblack: applying GRE MTU -> MSS fixup to cobalt and gerrit2001 - T218184
  • 11:41 Amir1: EU SWAT is done
  • 11:40 ladsgroup@deploy1001: Synchronized php-1.34.0-wmf.21/maintenance/getReplicaServer.php: SWAT: maintenance/getReplicaServer.php: Remove reference to long-deleted config var (T232268) (duration: 01m 04s)
  • 11:29 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable AMC Outreach modal (T231436) (duration: 01m 04s)
  • 11:15 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set item terms on write both up to Q10mio (T225055) (duration: 01m 03s)
  • 11:10 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: TR: set WikibaseTaintedReferencesEnabled true on labs wikidatawiki (T232191) (duration: 01m 03s)
  • 10:57 mobrovac: drop the wiktionary definition keyspace - T231361
  • 10:23 moritzm: removed roentgenium/tureis in Ganeti T224559
  • 10:18 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
  • 10:18 jmm@cumin2001: START - Cookbook sre.hosts.decommission
  • 10:17 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
  • 10:17 jmm@cumin2001: START - Cookbook sre.hosts.decommission
  • 10:01 jynus: stopping and upgrading db1074
  • 09:56 jynus: upgrading mariadb client libary on mariadb root clients
  • 09:46 jiji@deploy1001: Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 50% - T219150 (duration: 01m 03s)
  • 09:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a (duration: 12m 15s)
  • 09:32 mobrovac@deploy1001: Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3a
  • 09:32 mobrovac@deploy1001: Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3 (duration: 13m 18s)
  • 09:19 mobrovac@deploy1001: Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #3
  • 09:16 mobrovac@deploy1001: Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2 (duration: 03m 59s)
  • 09:13 mobrovac@deploy1001: Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints, take #2
  • 09:11 mobrovac@deploy1001: Finished deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449 (duration: 03m 24s)
  • 09:08 mobrovac@deploy1001: Started deploy [restbase/deploy@cf2ca76]: Stop using storage for enwiktionary definition and expose new PCS javascript endpoints - T231361 T232449
  • 08:36 mobrovac@deploy1001: Finished deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361 (duration: 02m 13s)
  • 08:34 mobrovac@deploy1001: Started deploy [changeprop/deploy@7a8ab89]: Stop pregenerating enwiktionary page/definition, take #2 - T231361
  • 08:24 mobrovac@deploy1001: Finished deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition (duration: 00m 34s)
  • 08:24 mobrovac@deploy1001: Started deploy [changeprop/deploy@069d297]: Revert Stop pregenerating enwiktionary page/definition
  • 08:22 mobrovac@deploy1001: Finished deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361 (duration: 02m 45s)
  • 08:19 mobrovac@deploy1001: Started deploy [changeprop/deploy@56a8342]: Stop pregenerating enwiktionary page/definition - T231361
  • 08:13 elukey: add thirdparty/amd-rocm271 to buster-wikimedia and update it with ROCm 2.7.1 packages
  • 08:09 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 08:07 elukey: execute reprepro clearvanished on install1002 to clear buster-wikimedia|thirdparty/amd-rocm27 (not used anymore)
  • 08:07 jmm@cumin1001: START - Cookbook sre.hosts.downtime
  • 08:04 marostegui@cumin1001: dbctl commit (dc=all): 'Fully repool db1122', diff saved to https://phabricator.wikimedia.org/P9088 and previous config saved to /var/cache/conftool/dbconfig/20190911-080450-marostegui.json
  • 07:52 moritzm: reimaging restbase-dev1005 to Stretch T224554
  • 07:51 marostegui@cumin1001: dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9087 and previous config saved to /var/cache/conftool/dbconfig/20190911-075139-marostegui.json
  • 07:33 marostegui@cumin1001: dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9086 and previous config saved to /var/cache/conftool/dbconfig/20190911-073335-marostegui.json
  • 07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9085 and previous config saved to /var/cache/conftool/dbconfig/20190911-072344-marostegui.json
  • 07:14 marostegui@cumin1001: dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9084 and previous config saved to /var/cache/conftool/dbconfig/20190911-071450-marostegui.json
  • 07:07 marostegui: Stop MySQL on db1122 to reboot for a kernel upgrade T230785
  • 07:06 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1122 to reboot for kernel upgrade T230785', diff saved to https://phabricator.wikimedia.org/P9083 and previous config saved to /var/cache/conftool/dbconfig/20190911-070635-marostegui.json
  • 07:00 hashar: Restarting Gerrit - T224448
  • 06:58 hashar: Restarting Gerrit
  • 06:45 marostegui: Drop unused database puppet on m1 - T231539
  • 06:19 marostegui@cumin1001: dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9082 and previous config saved to /var/cache/conftool/dbconfig/20190911-061924-marostegui.json
  • 06:17 marostegui@cumin1001: dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9081 and previous config saved to /var/cache/conftool/dbconfig/20190911-061659-marostegui.json
  • 05:48 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2048, will be decommissioned T230106', diff saved to https://phabricator.wikimedia.org/P9080 and previous config saved to /var/cache/conftool/dbconfig/20190911-054855-marostegui.json
  • 05:47 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db2112 to s1 codfw master T230106', diff saved to https://phabricator.wikimedia.org/P9079 and previous config saved to /var/cache/conftool/dbconfig/20190911-054753-marostegui.json
  • 05:29 marostegui: Switchover s1 codfw master db2048 -> db2112 T230106
  • 03:31 eileen: civicrm revision changed from b343642c76 to 53aeba6318, config revision is 3e22a80bc8

2019-09-10

  • 20:46 ejegg: updated payments-wiki from 15baf7f58b to 5432f9c3a4
  • 20:24 XioNoX: add MSS clamp on install1002 - T2324563
  • 20:20 XioNoX: add MSS clamp on archiva1001 - T232456
  • 18:42 herron: rolling out "Aggregate IPsec Tunnel Status” icinga check, please disregard for the time being if it alerts
  • 18:15 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T229863 Remove EventBusRCFeedEngine eventServiceName (duration: 01m 05s)
  • 18:15 XioNoX: rollback test add static route on bast3002 to force advmss
  • 18:10 XioNoX: test add static route on bast3002 to force advmss
  • 17:58 jforrester@deploy1001: Synchronized wmf-config/logging.php: T232042 Direct Parsoid/PHP rt-testing log events to a different target (duration: 01m 02s)
  • 17:56 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: T232122 Stop setting production value for eventlogging-service (duration: 01m 00s)
  • 17:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T232122 Remove use of eventlogging-service (duration: 01m 03s)
  • 17:33 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Re-sync for safety after scap errored with a broken pipe (duration: 01m 03s)
  • 17:31 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Variant configuration: Write to static (JSON) as well as serialised cache for testwiki T223602 (duration: 01m 02s)
  • 17:29 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: Variant configuration: Be able to write to static (JSON) as well as serialised cache (duration: 01m 03s)
  • 16:35 elukey: reboot analytics-tool1001 via ganeti gnt - not reachable via ssh
  • 16:24 urandom: disabling reserved space on restbase-dev1005:/dev/mapper/restbase--dev1005--vg-srv -- T224554
  • 16:10 marostegui: Failover m1 from db1063 to db1135 - T231403
  • 15:58 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert "Set items term store on write both for all of Wikidata" (duration: 01m 02s)
  • 15:58 thcipriani: restarting gerrit (again) https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?orgId=1&from=1568109359163&to=1568130959163&var-Application=&var-Window=30m due to T224448
  • 15:39 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.22
  • 15:37 marostegui: Start pre-switchover for m1 steps T231403
  • 15:35 hashar@deploy1001: Synchronized php-1.34.0-wmf.22/includes/libs/http/MultiHttpClient.php: Revert "Improve MultiHttpClient connection concurrency and reuse" - T232487 (duration: 00m 55s)
  • 15:33 reedy@deploy1001: Synchronized php-1.34.0-wmf.22/includes/libs/http/MultiHttpClient.php: T232487 (duration: 00m 55s)
  • 15:13 hashar@deploy1001: rebuilt and synchronized wikiversions files: Revert group0 to 1.34.0-wmf.22 # T220747
  • 14:48 hashar@deploy1001: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 14:45 akosiaris: repool cp1075 ats-be, releases cert updated
  • 14:44 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,dc=eqiad,cluster=cache_text,service=ats-be
  • 14:44 XioNoX: depool ulsfo for DC UPS power maintenance (see maint-announce)
  • 14:36 @: helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 14:32 hashar@deploy1001: Finished scap: testwiki to php-1.34.0-wmf.22 and rebuild l10n cache # T220747 (duration: 34m 03s)
  • 14:31 @: helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 14:29 @: helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 14:26 @: helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' .
  • 14:20 @: helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' .
  • 14:18 @: helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' .
  • 14:18 ottomata: increasing max_body_size to 10mb for all eventgate services - T232362
  • 14:14 akosiaris: depool cp1075 ats-be to test helmfile sync
  • 14:14 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,dc=eqiad,cluster=cache_text,service=ats-be
  • 13:58 hashar@deploy1001: Started scap: testwiki to php-1.34.0-wmf.22 and rebuild l10n cache # T220747
  • 13:56 hashar: Applied security patches to 1.34.0-wmf.22 # T220747
  • 13:53 hashar: scap prep 1.34.0-wmf.22 # T220747
  • 13:34 elukey: reboot stat1005 to clear incosistent process state after tensorflow tests
  • 13:23 hashar: ./make-wmf-branch -n 1.34.0-wmf.22 -o master -c extensions/CharInsert # T220747
  • 13:12 thcipriani: restarting gerrit
  • 13:11 hashar: Gerrit experimenting difficulty due to ongoing wmf branch cut - T231872
  • 13:01 moritzm: copied prometheus-jmx-exporter to buster-wikimedia (from stretch-wikimedia, just a package with some jars)
  • 12:40 cmjohnson1: the new pdus are racked in b6
  • 12:14 cmjohnson1: removing power from ps1-b6 side B...mgmt should not be affected
  • 11:20 cmjohnson1: swapping the PDU in rack B6 eqiad T227541
  • 11:09 Urbanecm: EU SWAT done
  • 11:09 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: c780fa4: Bump MobileWebUIActionsTracking sampling rate to 10 percent (T220016) (duration: 00m 55s)
  • 11:07 ema@puppetmaster1001: conftool action : set/weight=100; selector: service=ats-be,dc=eqiad,name=cp1075.eqiad.wmnet
  • 11:06 ema: cp1075: set weight in etcd back to 100
  • 11:06 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 6afe963: Set items term store on write both for all of Wikidata (T225055) (duration: 00m 55s)
  • 10:51 akosiaris@: helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 10:45 akosiaris@: helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'coredns' .
  • 10:45 akosiaris@: helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' .
  • 10:34 akosiaris@: helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 10:34 akosiaris@: helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'coredns' .
  • 10:34 akosiaris@: helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' .
  • 10:34 @: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' .
  • 10:34 @: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' .
  • 10:34 @: helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' .
  • 10:32 vgutierrez: repool cp5001 with ats-tls collecting memory usage details every hour - T232298
  • 09:56 elukey: restart archiva on archiva1001 - UI not working (probably due to connections to maven central being stuck)
  • 09:50 moritzm: installing ghostscript security updates on jessie
  • 09:37 moritzm: added jbond as chanserv ops for #wikimedia-operations
  • 08:08 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 08:06 jmm@cumin2001: START - Cookbook sre.hosts.downtime
  • 07:42 moritzm: reimaging mw2231 after hardware maintenance T231192
  • 07:21 moritzm: iron.wikimedia.org is no longer a bastion host
  • 06:57 moritzm: upgrading snapshot* to PHP 7.2.22 T230024
  • 05:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1073 from config T231892 (duration: 00m 54s)
  • 05:45 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1073 from config T231892 (duration: 00m 55s)
  • 05:35 marostegui: Stop MySQL on db2047 T231852
  • 05:35 marostegui: Remove db2047 from tendril and zarcillo - T231852
  • 05:33 urandom: decommissioning Cassandra, restbase-dev1005-b -- T224554
  • 05:15 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1104 into API T230762', diff saved to https://phabricator.wikimedia.org/P9071 and previous config saved to /var/cache/conftool/dbconfig/20190910-051529-marostegui.json
  • 05:02 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db1109 to s8 master and remove read-only from s8 T227062', diff saved to https://phabricator.wikimedia.org/P9070 and previous config saved to /var/cache/conftool/dbconfig/20190910-050213-marostegui.json
  • 05:00 marostegui@cumin1001: dbctl commit (dc=all): 'Set s8 as read-only for maintenance T230762', diff saved to https://phabricator.wikimedia.org/P9069 and previous config saved to /var/cache/conftool/dbconfig/20190910-050046-marostegui.json
  • 05:00 marostegui: Starting s8 failover from db1104 to db1109 - T227062
  • 04:46 vgutierrez: depool cp5001 for memory leak debugging on ATS - T232298
  • 04:23 marostegui: Start topology changes on s8, connect everything under db1109 - T230762
  • 04:22 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1109 with weight 0 and depool it from API T230762', diff saved to https://phabricator.wikimedia.org/P9068 and previous config saved to /var/cache/conftool/dbconfig/20190910-042243-marostegui.json
  • 04:18 marostegui: Start s8 (wikidata) pre switchover steps T230762
  • 00:59 krinkle@deploy1001: Finished deploy [performance/navtiming@f2a0863]: (no justification provided) (duration: 00m 05s)
  • 00:59 krinkle@deploy1001: Started deploy [performance/navtiming@f2a0863]: (no justification provided)
  • 00:57 Krinkle: krinkle@deploy1001: Deploy performance/navtiming f2a0863 - T226539
  • 00:41 urandom: decommissioning Cassandra, restbase-dev1005-a -- T224554

2019-09-09

  • 23:44 catrope@deploy1001: Synchronized php-1.34.0-wmf.21/skins/MinervaNeue/: T232260 (duration: 00m 57s)
  • 22:28 ejegg: updated payments-wiki from 51d9ed79b6 to 15baf7f58b
  • 20:50 urandom: bootstrapping Cassandra, restbase-dev1004-b -- T224554
  • 19:48 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 05m 45s)
  • 19:42 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9
  • 19:41 mdholloway: mobileapps deployment failed repooling canary (scb2001); retrying
  • 19:40 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9 (duration: 02m 59s)
  • 19:37 XioNoX: fix eqsin CF tunnel missconfig
  • 19:37 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@533d541]: Update mobileapps to 01971d9
  • 17:56 andrewbogott: disabling puppet on labpuppetmaster1001 as part of T171188
  • 17:55 XioNoX: push cloudflare tunnel config to cr1-eqsin
  • 16:50 papaul: replacing Fan kit and power supplies on cr1-codfw
  • 14:22 urandom: bootstrapping Cassandra, restbase-dev1004-a -- T224554
  • 13:51 vgutierrez: upgrading ats to 8.0.5-1wm6 on cp5001 - T232298
  • 13:39 vgutierrez: uploaded trafficserver 8.0.5-1wm6 to apt.wikimedia.org (stretch) - T232298
  • 13:31 moritzm: installing facter update from buster 10.1 point release (T222356)
  • 13:15 moritzm: upgrading labweb/wikitech to PHP 7.2.22 T230024
  • 13:02 Urbanecm: Patch is deployed, deploy1001 should be clear
  • 13:01 moritzm: upgrading remaining mediawiki app servers (mw1266-mw1275) to PHP 7.2.22 T230024
  • 12:55 urbanecm@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/WikibaseMediaInfo/: ubn patch T231276 (duration: 00m 58s)
  • 12:51 urbanecm@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/Wikibase: ubn patch T231276 (duration: 01m 03s)
  • 12:48 moritzm: upgrading remaining job runners to PHP 7.2.22 T230024
  • 12:44 Urbanecm: EU SWAT wmf patch ongoing, testing with mwdebug1002
  • 12:41 ema: lvs1015 (primary): restart pybal to add service restbase-ssl T210411
  • 12:36 ema: lvs2003 (primary): restart pybal to add service restbase-ssl T210411
  • 12:32 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: service=restbase-ssl,dc=eqiad
  • 12:30 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: service=restbase-ssl,dc=codfw
  • 12:29 elukey: restart archiva again to debug download artifact issue
  • 12:24 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: service=restbase-ssl,name=restbase2009.codfw.wmnet
  • 12:24 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: service=restbase-ssl,name=restbase1022.eqiad.wmnet
  • 12:11 Urbanecm: Undeployed patch in wmf branch, will resolve soon
  • 12:01 moritzm: installing ldap-corp1001 T231015
  • 11:32 Urbanecm: Dry run for all wikis (T231137)
  • 11:26 moritzm: installing ldap-corp2001 T231015
  • 10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 53s)
  • 10:38 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 54s)
  • 10:22 effie: jiji@deploy1001:~$ scap sync-file wmf-config/CommonSettings.php "Push PHP7 traffic to 33.3% - T219150"
  • 09:48 moritzm: updated stretch netinst image to 9.11 T232308
  • 09:42 eileen: civicrm revision changed from d1d65f37ea to 516eeb54b5, config revision is 5a6a9c6c03
  • 09:40 moritzm: updated buster netinst image to 10.1 T232310
  • 09:28 ema: lvs1016, lvs2006 (secondaries): restart pybal to add service restbase-ssl T210411
  • 09:02 elukey: restart archiva on archiva1001 - stuck and not serving requests (no trace about why in the logs)
  • 08:55 eileen: civicrm revision is d1d65f37ea, config revision is 5a6a9c6c03
  • 08:38 vgutierrez: disabling systemd hardening for ats-tls on cp5001 - T232298
  • 07:33 moritzm: installing ghostscript security updates
  • 03:53 vgutierrez: reboot analytics-tool1001
  • 02:59 bd808: Testing twitter integration after software update for Stashbot. In theory messages up to 280 characters in length will now be passed through to the @wikimediatech Twitter feed without being truncated. This message should end with a unicorn face if that is correct. 🦄

2019-09-08

2019-09-06

  • 21:33 cdanis: cdanis@mw1317.eqiad.wmnet ~ 🕠🍺 sudo -i depool
  • 21:27 James_F: mw1317 seems corrupted (Fatal error: Class undefined: stdClass in /srv/mediawiki/php-1.34.0-wmf.21/includes/libs/rdbms/database/DatabaseMysqli.php); running scap pull
  • 18:01 godog: silence esams pages for 30m
  • 17:43 crusnov@deploy1001: Finished deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux (duration: 02m 55s)
  • 17:40 crusnov@deploy1001: Started deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux
  • 17:39 crusnov@deploy1001: Finished deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux 3 (duration: 00m 21s)
  • 17:38 crusnov@deploy1001: Started deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux 3
  • 17:26 crusnov@deploy1001: Finished deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux 2 (duration: 00m 37s)
  • 17:25 crusnov@deploy1001: Started deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux 2
  • 17:25 crusnov@deploy1001: Finished deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux (duration: 01m 29s)
  • 17:24 crusnov@deploy1001: Started deploy [netbox/deploy@dea254a]: deploy for netbox split T223291 - buster redux
  • 14:56 jmm@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • 14:51 jmm@cumin2001: START - Cookbook sre.ganeti.makevm
  • 14:48 jmm@cumin2001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • 14:43 jmm@cumin2001: START - Cookbook sre.ganeti.makevm
  • 12:38 ema: cp5001: restart trafficserver-tls.service to clear icinga alert after segfault
  • 12:36 moritzm: fix permissions on /var/spool/exim on krypton (hosts used to run the exim heavy role which uses different permissions than the light role)
  • 10:59 onimisionipe: force shard allocation - chi eqiad
  • 10:59 Amir1: ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=testwikidatawiki (T225056)
  • 10:17 moritzm: installing exim4 security updates
  • 08:43 mutante: webperf* - /usr/local/sbin/build-envoy-config -c /etc/envoy | rm /etc/envoy/listeners.d/00-tls_terminator_443.yaml | run puppet - envoy now listening on 443 (T210411)
  • 07:48 mutante: running puppet on cp-text_eqiad / cp1075 - switching releases.wikimedia.org to TLS to backend
  • 06:29 oblivian@deploy1001: Synchronized README: testing php conditional restarts (duration: 00m 55s)
  • 06:09 mutante: puppetmaster1001 - same for restbase-dev1005 and restbase-dev1006 (T224554)
  • 06:03 mutante: puppetmaster1001 - copying cassandra-ca-manager to /usr/local/bin - deleting expired restbase-dev1004 certs - running cassandra-ca-manager services-dev.yaml T224554
  • 05:31 marostegui: Stop MySQL on db2046 - T231767
  • 05:11 marostegui: Remove db2046 from tendril and zarcillo - T231767
  • 04:54 _joe_: run systemctl reset-failed on kafka1001 to clear a 13 hours icinga alert
  • 03:21 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: deploy for netbox split T223291 (duration: 00m 14s)
  • 03:21 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: deploy for netbox split T223291
  • 03:16 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: deploy for netbox split T223291 (testing) (duration: 00m 20s)
  • 03:16 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: deploy for netbox split T223291 (testing)
  • 03:07 chaomodus: restarting keyholder on deploy1001
  • 02:34 ejegg: rolled back payments-wiki to 51d9ed79b6
  • 02:25 ejegg: updated payments-wiki (again) from 51d9ed79b6 to 04120169b0... false alarm
  • 02:15 ejegg: payments-wiki rolled back to 51d9ed79b6
  • 02:11 ejegg: updated payments-wiki from 51d9ed79b6 to 04120169b0
  • 01:44 eileen: tools revision changed from 643c48b26a to 1e405864d7
  • 01:18 ayounsi@deploy1001: Finished deploy [netbox/deploy@367ca84]: test (duration: 00m 02s)
  • 01:18 ayounsi@deploy1001: Started deploy [netbox/deploy@367ca84]: test

2019-09-05

  • 23:13 ayounsi@deploy1001: Finished deploy [netbox/deploy@367ca84]: test (duration: 00m 42s)
  • 23:12 ayounsi@deploy1001: Started deploy [netbox/deploy@367ca84]: test
  • 23:09 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T151425 Require that passwords are not in the most common 100k list for all users (duration: 00m 48s)
  • 22:12 eileen: tools revision changed from b42bda6bf3 to 643c48b26a
  • 21:42 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: deploy for netbox split T223291 (duration: 00m 03s)
  • 21:42 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: deploy for netbox split T223291
  • 21:35 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: test deploy for netbox split - again (duration: 00m 12s)
  • 21:34 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: test deploy for netbox split - again
  • 19:28 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: c7678f0e3d638 (duration: 00m 47s)
  • 19:21 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/WikimediaMaintenance/blameStartupRegistry.php: 7adf466614d (duration: 00m 48s)
  • 18:10 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: test deploy for netbox split (duration: 38m 39s)
  • 17:31 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: test deploy for netbox split
  • 16:22 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all events to eventgate - T228705 - take 2 (duration: 00m 49s)
  • 16:06 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all events to eventgate - T228705 (duration: 00m 48s)
  • 16:04 ottomata: switching remaining job queue events (and all remaining events) to eventgate - T228705
  • 15:59 jynus: restarting batch processes on mwmaint1002 T232106
  • 15:54 jynus@deploy1001: Synchronized private/PrivateSettings.php: updating cli password (duration: 00m 47s)
  • 15:23 herron: beginning replacement of kafka1001 with kafka-main1001 T225005
  • 14:54 ema: restbase2009: repool after successful envoy deployment T210411
  • 14:50 ema: restbase2009: depool and add TLS termination w/ envoy -- https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/533028/ T210411
  • 14:42 XioNoX: remove iron from mr* routers - T231811
  • 14:30 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=prometheus1003.eqiad.wmnet
  • 14:15 @: helmfile [EQIAD] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
  • 14:14 @: helmfile [CODFW] Ran 'sync' command on namespace 'sessionstore' for release 'production' .
  • 14:11 cdanis: restarted swiftrepl on ms-fe1005 T231110
  • 13:54 @: helmfile [CODFW] Ran 'apply' command on namespace 'sessionstore' for release 'production' .
  • 13:39 moritzm: upgrading remaining API servers to PHP 7.2.22
  • 13:37 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 13:21 filippo@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=prometheus1003.eqiad.wmnet
  • 13:17 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=prometheus1004.eqiad.wmnet
  • 13:04 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.21
  • 12:47 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 12:13 moritzm: upgrading mw1284-mw1290 to PHP 7.2.22
  • 12:02 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 11:57 moritzm: upgrading remaining job runners to PHP 7.2.22
  • 11:50 dcausse: EU swat done
  • 11:48 dcausse@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/CirrusSearch/: T159321: Add morelikethis a non-greedy version of the morelike keyword (duration: 00m 59s)
  • 10:53 godog: temporarily enable prometheus admin web api in prometheus@ops in eqiad to delete spammy metrics - T228395
  • 10:49 filippo@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=prometheus1004.eqiad.wmnet
  • 10:46 moritzm: upgrading mw1221-mw1335 to PHP 7.2.22
  • 10:31 moritzm: upgrading mw1319-mw1333 to PHP 7.2.22
  • 10:28 _joe_: upgrading scap across the fleet T224857
  • 10:25 moritzm: upgrading mw1238-mw1258 to PHP 7.2.22
  • 09:39 mutante: ganeti1001 - creating VM moscovium (T232077)
  • 09:26 vgutierrez: rolling back from ats-tls to nginx on cp1076 - T231433
  • 09:17 jmm@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 09:17 jmm@cumin2001: START - Cookbook sre.hosts.downtime
  • 09:05 hashar@deploy1001: rebuilt and synchronized wikiversions files: Promote wikidatawiki to 1.34.0-wmf.21 for T232035 - T220746
  • 09:04 vgutierrez: rolling back from ats-tls to nginx on cp3034 - T231433
  • 08:55 hashar@deploy1001: rebuilt and synchronized wikiversions files: Rollback wikidatawiki to 1.34.0-wmf.20 for T232035
  • 08:38 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=a.*-ro,name=codfw
  • 08:37 jmm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 08:35 jmm@cumin1001: START - Cookbook sre.hosts.downtime
  • 08:32 akosiaris: depool restbase1022 T232007
  • 08:30 vgutierrez: rebooting cp3034
  • 08:23 vgutierrez: repooling cp3034
  • 08:21 hashar@deploy1001: rebuilt and synchronized wikiversions files: Promote wikidatawiki to 1.34.0-wmf.21 for T232035 - T220746
  • 08:16 moritzm: reimage restbase-dev1004 to Stretch T224554
  • 08:13 _joe_: upgrading scap on deploy1001
  • 08:09 vgutierrez: depooling cp3034 due to intermittent network issues
  • 07:57 _joe_: upgrading scap on mwdebug1001
  • 07:56 _joe_: uploading scap 3.12.1 to reprepro on all distros 224857
  • 07:56 hashar: Switching "wikidatawiki" on mwdebug1001 to 1.34.0-wmf.21 by editing /srv/mediawiki/wikiversions.php # T232035
  • 07:53 marostegui: Remove old backups for db2037 and db2042 from dbprov2001
  • 07:45 marostegui: Remove puppet grants from m1 for the following IPs: 10.64.0.165 10.64.16.159 10.64.16.18 T231539
  • 07:32 moritzm: upgrading mw1293-mw1296, mw1299-mw1306 to PHP 7.2.22
  • 07:31 mutante: ununpentium - removed /etc/envoy/envoy.yaml; ran /usr/local/sbin/build-envoy-config -c /etc/envoy to regenarate config without 443 listener; ran puppet; envoy now running on jessie
  • 07:07 mutante: ununpentium - manually delete /etc/envoy/listeners.d/00-tls_terminator_443.yaml after changing port to 1443 - puppet does not remove it
  • 06:44 kart_: Updated cxserver to 2019-09-04-065911-production (T213255, T206310)
  • 06:41 @: helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' .
  • 06:39 @: helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' .
  • 06:38 @: helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' .
  • 05:42 marostegui: Remove grants for dbproxy1005 T231280 T231967
  • 05:31 marostegui: Restart MySQL on codfw sanitariums (db1124 and db1125) to pick up new filters - T51195
  • 05:29 marostegui: Restart wikibugs
  • 05:21 mutante: ganeti2005 - DRAC reset fails - ipmi_cmd_cold_reset: bad completion code
  • 05:19 mutante: ganeti2005 - reset DRAC via local IPMI since mgmt stopped responding
  • 05:14 marostegui: Restart MySQL on codfw sanitariums (db2094 and db2095) to pick up new filters - T51195
  • 04:57 vgutierrez: rearming keyholder on cumin1001
  • 04:42 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp4021 - T231433
  • 04:37 vgutierrez: switching cp4021 from nginx to ats-tls - T231433
  • 04:31 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp3034 - T231433
  • 04:20 vgutierrez: switching cp3034 from nginx to ats-tls - T231433
  • 04:02 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp1076 - T231433
  • 03:57 vgutierrez: switching cp1076 from nginx to ats-tls - T231433
  • 00:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 53s)
  • 00:54 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 1 (duration: 00m 56s)
  • 00:04 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out load of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 55s)
  • 00:02 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out load of variant config into MWConfigCacheGenerator, part 1 (duration: 00m 55s)

2019-09-04

  • 23:36 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out variant config generation into MWConfigCacheGenerator, part 2 (duration: 00m 55s)
  • 23:33 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out variant config generation into MWConfigCacheGenerator, part 1 (duration: 00m 54s)
  • 23:05 urandom: decommission restbase-dev1004-b (Cassandra) -- T224554
  • 21:58 andrewbogott: attached to console on cumin1001, found it in bios 'system settings', exited, allowed boot to continue. No idea how it got there — spontaneous reboot?
  • 21:12 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: (no justification provided) (duration: 08m 55s)
  • 21:03 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: (no justification provided)
  • 20:14 urandom: decommission restbase-dev1004-a (Cassandra) -- T224554
  • 20:00 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 19:35 hashar@deploy1001: rebuilt and synchronized wikiversions files: rollback wikidatawiki to 1.34.0-wmf.20 for T232035 - T220746
  • 19:33 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 19:17 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 19:00 hashar@deploy1001: Synchronized php: group1 wikis to 1.34.0-wmf.21 (duration: 00m 54s)
  • 18:59 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.21
  • 17:59 jforrester@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/modules/homepage/: T229271 Homepage: Unbreak question dialogs on mobile (duration: 00m 56s)
  • 17:47 jforrester@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T150418 Fix HTML blacklist inheritance to avoid copy-pasted read <ref>s again (duration: 00m 57s)
  • 17:45 jforrester@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T150418 Fix HTML blacklist inheritance to avoid copy-pasted read <ref>s again (duration: 00m 56s)
  • 17:43 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 - take 2 (duration: 00m 55s)
  • 17:34 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 (duration: 00m 56s)
  • 17:32 ottomata: Switch all non-low-traffic jobs to eventgate - T228705
  • 17:14 @: helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 16:50 @: helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 16:48 joal@deploy1001: Finished deploy [analytics/refinery@2322f10]: Fix for yesterday regular analytics deploy (duration: 53m 16s)
  • 16:40 Lucas_WMDE: Morning SWAT done
  • 16:38 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/AbuseFilter: SWAT: Fix filter validation in ViewEdit (T231985) (duration: 00m 58s)
  • 16:11 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 533172|Move ContentTranslation out of Beta in jvwiki (T231207) (duration: 00m 56s)
  • 15:55 joal@deploy1001: Started deploy [analytics/refinery@2322f10]: Fix for yesterday regular analytics deploy
  • 15:36 godog: upgrade grafana to 5.4.5 on labmon
  • 14:51 andrewbogott: reimaging cloudvirt1015 for T220853
  • 14:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove obsoleted DB config from db-eqiad.php T231642 (duration: 00m 57s)
  • 14:08 cdanis: If0dd79604 actually live on canaries now
  • 14:04 cdanis: If0dd79604 deployed to eqiad MW canaries T231642
  • 13:59 moritzm: installing nghttp2 security updates
  • 13:59 cdanis: manually testing If0dd79604 on mwdebug1001
  • 13:47 _joe_: restarting php7.2-fpm across the fleet to pick up the apc.ttl removal
  • 13:20 cdanis@deploy1001: Synchronized wmf-config/db-codfw.php: a8dc4c4a0 db-codfw: remove obsoleted DB config T231642 (duration: 00m 55s)
  • 13:20 oblivian@cumin1001: END (PASS) - Cookbook sre.mediawiki.restart-appservers (exit_code=0)
  • 13:17 oblivian@cumin1001: START - Cookbook sre.mediawiki.restart-appservers
  • 13:17 oblivian@cumin1001: END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99)
  • 13:17 oblivian@cumin1001: START - Cookbook sre.mediawiki.restart-appservers
  • 12:56 cdanis: manually testing I1bc6d1603 on mwdebug2002
  • 12:49 gehel: reset kartotherian password on maps slaves - T231964
  • 12:36 gehel: restart kartotherian on maps1001 - T231964
  • 11:52 dcausse: EU SWAT done
  • 11:49 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T231194: [cirrus] Reenable sanity checks (duration: 00m 56s)
  • 11:47 dcausse@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/CirrusSearch/: T159321: Add morelikethis a non-greedy version of the morelike keyword (duration: 00m 57s)
  • 11:47 Amir1: start of ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 2000000 --sleep 2 > ~/rebuildItemTerms.out 2> rebuildItemTerms.err (T225056). This is going to take a while. On screen
  • 11:38 moritzm: upgrading mw1339-mw1348 to PHP 7.2.22
  • 11:37 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m (T225055) (duration: 00m 55s)
  • 11:32 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add high-density logos for the Incubator (T230122) (duration: 00m 56s)
  • 11:28 ladsgroup@deploy1001: Synchronized static/images/project-logos/incubatorwiki-2x.png: SWAT: Add high-density logos for the Incubator (T230122) Part II (duration: 00m 54s)
  • 11:27 ladsgroup@deploy1001: Synchronized static/images/project-logos/incubatorwiki-1.5x.png: SWAT: Add high-density logos for the Incubator (T230122) Part I (duration: 00m 52s)
  • 11:24 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf '%s\n' 'https://en.wikipedia.org/static/images/project-logos/wikidatawiki-1.5x.png' | mwscript purgeList.php wikidatawiki # T230120
  • 11:18 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add high-density logos for Wikidata (T230120) (duration: 00m 55s)
  • 11:14 ladsgroup@deploy1001: Synchronized static/images/project-logos/wikidatawiki-2x.png: SWAT: Add high-density logos for Wikidata (T230120) Part II (duration: 00m 56s)
  • 11:12 ladsgroup@deploy1001: Synchronized static/images/project-logos/wikidatawiki-1.5x.png: SWAT: Add high-density logos for Wikidata (T230120) Part I (duration: 00m 56s)
  • 10:42 marostegui: Start event scheduler on db1115 T231769
  • 10:23 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp2002 - T231859
  • 10:20 marostegui: Start MySQL on db1115 without the event scheduler - T231769
  • 10:12 marostegui: Stop MySQL on db1115 without the event scheduler - T231769
  • 10:12 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp5001 - T231859
  • 10:11 @: helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' .
  • 10:11 marostegui: Tendril/dbtree will be unavailable for a few minutes T231769
  • 10:11 marostegui: Stop MySQL on db1115 - T231769
  • 10:09 vgutierrez: uploaded trafficserver 8.0.5-1wm5 to apt.wikimedia.org (stretch) - T231533 T231859
  • 09:33 moritzm: upgrading mw servers in codfw to 7.2.22
  • 09:19 _joe_: uploaded envoyproxy to buster
  • 08:56 moritzm: upgrading mw1238-mw1258 to PHP 7.2.22
  • 08:42 marostegui: Stop HAproxy on dbproxy1005 - T231967
  • 08:37 moritzm: upgrading API canaries in eqiad to 7.2.22
  • 08:26 marostegui: Reboot db1135 to pick up new kernel - T231403
  • 07:50 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db2047 from config T231852 (duration: 00m 54s)
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db2047 from config T231852 (duration: 00m 57s)
  • 07:21 mutante: ununpentium - a2dismod ssl - systemctl restart apache2
  • 05:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 05:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
  • 02:46 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/resources/src/startup/mediawiki.js: 8a1b13026 (duration: 00m 55s)
  • 02:42 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/resources/src/mediawiki.base/mediawiki.base.js: 8a1b13026 (duration: 00m 56s)
  • 02:21 chaomodus: extending downtime on netmon1002 and netmon2001, netbox1001, netbox2001, netboxdb1001 and netbox2001 should be stable but are still being debugged
  • 01:02 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s)
  • 00:02 chaomodus: installing and setting up netbox instances T223291

2019-09-03

  • 23:57 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s)
  • 23:56 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s)
  • 23:54 niharika29@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 55s)
  • 23:52 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 56s)
  • 23:42 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/tests/phpunit/: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 57s)
  • 23:41 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/includes/block: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 55s)
  • 23:28 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable and configure ORES damaging and goodfaith on zhwiki T225562 (duration: 00m 58s)
  • 23:10 ebernhardson: production-search-eqiad all indices index.merge.policy.deletes_pct_allowed=20
  • 22:54 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208694 Set CentralNotice's wgNoticeProjects for wikimedia (duration: 00m 59s)
  • 22:45 eileen: process-control config revision is 100334de4a adjust silverpop schedule
  • 19:42 XioNoX: rollback OSPF metric change on eqiad-codfw Zayo link (1320->320)
  • 19:20 fdans@deploy1001: Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided)
  • 19:18 fdans@deploy1001: Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided)
  • 19:14 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch high-traffic jobs to eventgate. Take 2 - T228705 (duration: 00m 56s)
  • 19:12 ottomata: switching jobqueue events to eventgate-main - T228705
  • 18:41 urbanecm@deploy1001: Synchronized wmf-config/: Emergency fix: GE not loading configuration properly: newbie facing feature (duration: 00m 57s)
  • 18:35 Urbanecm: Livetesting on mwdebug1002
  • 17:45 James_F: Pulled I9b64a2bb770 into wmf.21 production on the deploy server; no need to deploy to app-servers, CI-only fix.
  • 17:40 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21
  • 16:35 catrope@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/Graph/includes/ApiGraph.php: T231894 (duration: 00m 55s)
  • 16:01 joal@deploy1001: Finished deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy (duration: 136m 59s)
  • 15:55 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T227260 (duration: 00m 54s)
  • 15:32 ebernhardson: unban elastic1027 from production-search-eqiad
  • 15:07 hashar@deploy1001: rebuilt and synchronized wikiversions files: testwiki 1.34.0-wmf.21 for T231894 - T220746
  • 14:57 hashar@deploy1001: rebuilt and synchronized wikiversions files: Rollback group0 to 1.34.0-wmf.21 - T220746
  • 14:45 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21 - T220746
  • 14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Promote db1133 as wikitech master T229657 (duration: 00m 54s)
  • 14:28 hashar@deploy1001: Finished scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 (duration: 50m 09s)
  • 14:21 moritzm: upgrading app server canaries to PHP 7.2.22 T230024
  • 13:44 joal@deploy1001: Started deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy
  • 13:38 hashar@deploy1001: Started scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746
  • 13:26 hashar: Gerrit should be fine again, apparently was due to the wmf branch cut taking too much resources (sic) - T231872 filled to investigate
  • 13:25 hashar: 1.34.0-wmf.21 cut
  • 13:16 hashar: Gerrit has some random times out from time to time (no reason)
  • 13:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1073 from wikitech T229657', diff saved to https://phabricator.wikimedia.org/P9038 and previous config saved to /var/cache/conftool/dbconfig/20190903-131456-marostegui.json
  • 13:13 marostegui: Re-enable puppet on db1073 and db1133 T229657
  • 13:11 marostegui: Reload haproxy on dbproxy1005 T229657
  • 13:10 marostegui@cumin1001: dbctl commit (dc=all): 'Set wikitech back to RW after maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9037 and previous config saved to /var/cache/conftool/dbconfig/20190903-131000-marostegui.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9033 and previous config saved to /var/cache/conftool/dbconfig/20190903-130113-marostegui.json
  • 13:00 marostegui: Failover m5 from db1073 to db1133 - T229657
  • 12:52 moritzm: uploaded PHP 7.2.22 to component/php72 T230024
  • 12:39 moritzm: upgrading mwdebug2001 to PHP 7.2.22
  • 12:29 hashar: Cutting wmf/1.34.0-wmf.21 # T220746
  • 12:19 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.20
  • 12:02 marostegui: Disable puppet on db1073 and db1133 - T229657
  • 11:55 marostegui: Change topology on m5 and make everything replicate from db1133 - T229657
  • 11:48 marostegui: Downtime m5 hosts T229657
  • 11:35 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 (T225056)
  • 11:29 Amir1: EU SWAT is done
  • 11:29 Amir1: ladsgroup@mwmaint1002:~$ mwscript namespaceDupes.php bswiki --fix (T231654)
  • 11:28 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix wgMetaNamespaceTalk for bswiki (T231654) (duration: 00m 54s)
  • 11:25 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016) (duration: 00m 52s)
  • 11:11 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016) (duration: 00m 53s)
  • 11:07 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WRITE_BOTH for items term store for wikidatawiki (T225055) (duration: 00m 55s)
  • 10:17 ema: cp1083: varnish-backend-restart -- mbox lag, fetch failures
  • 09:59 _joe_: removing old lvs-related scripts from ores*
  • 09:46 moritzm: moved uid=smalyshev from cn=wmf to cn=nda
  • 09:46 mutante: install1002 - import GPG key for getenvoy repo, importing envoy for jessie with reprepro update
  • 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
  • 09:04 ema: cp1085: varnish-backend-restart, mbox lag and fetch failures
  • 09:03 gehel: reset kartotherian password -T231842
  • 08:54 ema: cp1089: varnish-backend-restart due to mbox lag and fetch failures
  • 08:49 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be
  • 08:49 ema: cp1075: pool ats-be with caching enabled T228629
  • 08:26 marostegui: Add REPLICATION grant to wikiuser and wikiadmin on db1073 with replication enabled - T229657
  • 08:21 gehel: purging maps / info.json from cache - T231842
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9031 and previous config saved to /var/cache/conftool/dbconfig/20190903-080958-marostegui.json
  • 08:04 joal@deploy1001: Finished deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try (duration: 00m 27s)
  • 08:03 joal@deploy1001: Started deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try
  • 08:02 joal@deploy1001: deploy aborted: Regular weekly analytics deploy train (duration: 27m 47s)
  • 07:16 marostegui: Change min_replicas to 6 on s1 for eqiad and codfw T231019
  • 06:39 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9029 and previous config saved to /var/cache/conftool/dbconfig/20190903-063932-marostegui.json
  • 06:10 mutante: running puppet on cp-text_eqiad to switch people.wm.org to https backend
  • 06:04 marostegui: Change min_replicas to 4 on s7 for eqiad and codfw T231019
  • 05:53 mutante: people.wikimedia.org - switching to TLS termination with envoy
  • 05:52 marostegui@cumin1001: dbctl commit (dc=all): 'Reorganize s7 codfw T230106', diff saved to https://phabricator.wikimedia.org/P9028 and previous config saved to /var/cache/conftool/dbconfig/20190903-055234-marostegui.json
  • 05:47 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Reorganize s7 codfw T230106 (duration: 00m 54s)
  • 05:22 marostegui: Rename tables on the puppet database on m1 master - T231539
  • 05:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106 (duration: 00m 54s)
  • 05:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2047 old master from s7 T230106', diff saved to https://phabricator.wikimedia.org/P9027 and previous config saved to /var/cache/conftool/dbconfig/20190903-051619-marostegui.json
  • 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db2118 to s7 codfw master (db2047 -> db2118) T230106', diff saved to https://phabricator.wikimedia.org/P9026 and previous config saved to /var/cache/conftool/dbconfig/20190903-051450-marostegui.json
  • 05:02 marostegui: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106
  • 04:50 marostegui: Drop filejournal table on s3 - T51195
  • 04:49 vgutierrez: repooling cp2002 - T231433
  • 04:36 vgutierrez: upgrading ATS to 8.0.5-1wm4 on cp2002 - T231433
  • 04:28 vgutierrez: Switching cp2002 from nginx to ats-tls - T231433

2019-09-02

  • 22:08 ebernhardson: ban elastic1027 from production-search-chi
  • 20:48 ebernhardson: restart production-search-eqiad on elastic1027 again
  • 20:33 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@453ee8a]: Make osm-pbf source private (T231842) (duration: 02m 09s)
  • 20:31 mbsantos@deploy1001: Started deploy [kartotherian/deploy@453ee8a]: Make osm-pbf source private (T231842)
  • 19:54 ebernhardson: restart elasticsearch_6@production-search-eqiad on elastic1027
  • 17:57 mateusbs17: regenerating tiles from z0 to z9 in eqiad and codfw- T231691, T230511
  • 15:08 moritzm: installing libssh2 security updates
  • 14:36 moritzm: installing ghostscript updates on thumbor1001
  • 14:24 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 14:21 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 14:10 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 13:44 akosiaris: resync the sessionstore staging release as there was wrong port mapping (port 8080 instead of 8081) for both netpol and service
  • 13:43 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 13:40 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 13:09 vgutierrez: upgrading prometheus-trafficserver-exporter to version 0.3.2 on the cache cluster - T231533
  • 12:58 vgutierrez: upgrading prometheus-trafficserver-exporter to version 0.3.2 on cp5001 - T231533
  • 12:46 vgutierrez: uploaded prometheus-trafficserver-exporter 0.3.2 to apt.wikimedia.org (stretch) - T231533
  • 12:40 moritzm: installing freetype security updates on jessie (stretch/buster already fixed)
  • 11:23 moritzm: installing apache2 security updates on jessie
  • 11:18 moritzm: imported apache2 2.4.10-10+deb8u15+wmf1 to apt.wikimedia.org/jessie-wikimedia (rebuild of latest Jessie update against our patches)
  • 10:25 moritzm: installing libav security updates
  • 10:07 moritzm: installing subversion security updates on jessie
  • 09:21 marostegui: Drop filejournal table on s7 - T51195
  • 09:15 marostegui: Drop filejournal table on s1 - T51195
  • 08:45 marostegui: Drop filejournal table on s8 - T51195
  • 08:27 marostegui: Drop filejournal table on labtestwiki - T51195
  • 08:25 marostegui: Drop filejournal table on s2 - T51195
  • 08:15 godog: upgrade grafana to 5.4.5 on grafana1001
  • 08:12 godog: update amd-rocm debian repository gpg key (same id, new expiration)
  • 07:34 marostegui: Drop filejournal table on s4 - T51195
  • 07:26 marostegui: Drop filejournal table on s5 - T51195
  • 07:17 marostegui: Drop filejournal table on s6 - T51195
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db2046 from config T231767 (duration: 00m 53s)
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db2046 from config T231767 (duration: 00m 55s)

2019-09-01

  • 17:53 Urbanecm: Run mwscript extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --wiki=enwikiquote --verbose (T231137)
  • 17:45 Urbanecm: Run mwscript extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --wiki=metawiki --verbose (T231137)
  • 17:33 Urbanecm: Run foreachwikiindblist group1.dblist extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --dry-run --verbose (T231137)
  • 17:29 Urbanecm: Previous should be *group0.dblist (T231137)
  • 17:29 Urbanecm: Run foreachwikiindblist group0 extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --dry-run --verbose (T231137)


Archives

See Server admin log/Archives.