You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s))
imported>Stashbot
(jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 53s))
Line 1: Line 1:
== 2019-09-05 ==
* 00:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 53s)
* 00:54 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 1 (duration: 00m 56s)
* 00:04 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out load of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 55s)
* 00:02 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out load of variant config into MWConfigCacheGenerator, part 1 (duration: 00m 55s)
== 2019-09-04 ==
== 2019-09-04 ==
* 23:36 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out variant config generation into MWConfigCacheGenerator, part 2 (duration: 00m 55s)
* 23:33 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out variant config generation into MWConfigCacheGenerator, part 1 (duration: 00m 54s)
* 23:05 urandom: decommission restbase-dev1004-b (Cassandra) -- [[phab:T224554|T224554]]
* 21:58 andrewbogott: attached to console on cumin1001, found it in bios 'system settings', exited, allowed boot to continue.  No idea how it got there — spontaneous reboot?
* 21:12 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: (no justification provided) (duration: 08m 55s)
* 21:03 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: (no justification provided)
* 20:14 urandom: decommission restbase-dev1004-a (Cassandra) -- [[phab:T224554|T224554]]
* 20:00 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
* 19:35 hashar@deploy1001: rebuilt and synchronized wikiversions files: rollback wikidatawiki to 1.34.0-wmf.20 for [[phab:T232035|T232035]] - [[phab:T220746|T220746]]
* 19:33 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
* 19:17 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
* 19:00 hashar@deploy1001: Synchronized php: group1 wikis to 1.34.0-wmf.21 (duration: 00m 54s)
* 18:59 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.21
* 17:59 jforrester@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/modules/homepage/: [[phab:T229271|T229271]] Homepage: Unbreak question dialogs on mobile (duration: 00m 56s)
* 17:47 jforrester@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: [[phab:T150418|T150418]] Fix HTML blacklist inheritance to avoid copy-pasted read <ref>s again (duration: 00m 57s)
* 17:45 jforrester@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: [[phab:T150418|T150418]] Fix HTML blacklist inheritance to avoid copy-pasted read <ref>s again (duration: 00m 56s)
* 17:43 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - [[phab:T228705|T228705]] - take 2 (duration: 00m 55s)
* 17:34 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - [[phab:T228705|T228705]] (duration: 00m 56s)
* 17:32 ottomata: Switch all non-low-traffic jobs to eventgate - [[phab:T228705|T228705]]
* 17:14 @: helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
* 16:50 @: helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
* 16:48 joal@deploy1001: Finished deploy [analytics/refinery@2322f10]: Fix for yesterday regular analytics deploy (duration: 53m 16s)
* 16:40 Lucas_WMDE: Morning SWAT done
* 16:38 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/AbuseFilter: SWAT: [[gerrit:534429{{!}}Fix filter validation in ViewEdit (T231985)]] (duration: 00m 58s)
* 16:11 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit{{!}}533172{{!}}Move ContentTranslation out of Beta in jvwiki (T231207)]] (duration: 00m 56s)
* 15:55 joal@deploy1001: Started deploy [analytics/refinery@2322f10]: Fix for yesterday regular analytics deploy
* 15:36 godog: upgrade grafana to 5.4.5 on labmon
* 14:51 andrewbogott: reimaging cloudvirt1015 for [[phab:T220853|T220853]]
* 14:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove obsoleted DB config from db-eqiad.php [[phab:T231642|T231642]] (duration: 00m 57s)
* 14:08 cdanis: {{Gerrit|If0dd79604}} actually live on canaries now
* 14:04 cdanis: {{Gerrit|If0dd79604}} deployed to eqiad MW canaries [[phab:T231642|T231642]]
* 13:59 moritzm: installing nghttp2 security updates
* 13:59 cdanis: manually testing {{Gerrit|If0dd79604}} on mwdebug1001
* 13:47 _joe_: restarting php7.2-fpm across the fleet to pick up the apc.ttl removal
* 13:20 cdanis@deploy1001: Synchronized wmf-config/db-codfw.php: {{Gerrit|a8dc4c4a0}} db-codfw: remove obsoleted DB config [[phab:T231642|T231642]] (duration: 00m 55s)
* 13:20 oblivian@cumin1001: END (PASS) - Cookbook sre.mediawiki.restart-appservers (exit_code=0)
* 13:17 oblivian@cumin1001: START - Cookbook sre.mediawiki.restart-appservers
* 13:17 oblivian@cumin1001: END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99)
* 13:17 oblivian@cumin1001: START - Cookbook sre.mediawiki.restart-appservers
* 12:56 cdanis: manually testing {{Gerrit|I1bc6d1603}} on mwdebug2002
* 12:49 gehel: reset kartotherian password on maps slaves - [[phab:T231964|T231964]]
* 12:36 gehel: restart kartotherian on maps1001 - [[phab:T231964|T231964]]
* 11:52 dcausse: EU SWAT done
* 11:49 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: [[phab:T231194|T231194]]: [cirrus] Reenable sanity checks (duration: 00m 56s)
* 11:47 dcausse@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/CirrusSearch/: [[phab:T159321|T159321]]: Add morelikethis a non-greedy version of the morelike keyword (duration: 00m 57s)
* 11:47 Amir1: start of ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id {{Gerrit|2000000}} --sleep 2 > ~/rebuildItemTerms.out 2> rebuildItemTerms.err ([[phab:T225056|T225056]]). This is going to take a while. On screen
* 11:38 moritzm: upgrading mw1339-mw1348 to PHP 7.2.22
* 11:37 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:534183{{!}}Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m (T225055)]] (duration: 00m 55s)
* 11:32 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:534197{{!}}Add high-density logos for the Incubator (T230122)]] (duration: 00m 56s)
* 11:28 ladsgroup@deploy1001: Synchronized static/images/project-logos/incubatorwiki-2x.png: SWAT: [[gerrit:534197{{!}}Add high-density logos for the Incubator (T230122)]] Part II (duration: 00m 54s)
* 11:27 ladsgroup@deploy1001: Synchronized static/images/project-logos/incubatorwiki-1.5x.png: SWAT: [[gerrit:534197{{!}}Add high-density logos for the Incubator (T230122)]] Part I (duration: 00m 52s)
* 11:24 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf '%s\n' 'https://en.wikipedia.org/static/images/project-logos/wikidatawiki-1.5x.png' {{!}} mwscript purgeList.php wikidatawiki # [[phab:T230120|T230120]]
* 11:18 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:534152{{!}}Add high-density logos for Wikidata (T230120)]] (duration: 00m 55s)
* 11:14 ladsgroup@deploy1001: Synchronized static/images/project-logos/wikidatawiki-2x.png: SWAT: [[gerrit:534151{{!}}Add high-density logos for Wikidata (T230120)]] Part II (duration: 00m 56s)
* 11:12 ladsgroup@deploy1001: Synchronized static/images/project-logos/wikidatawiki-1.5x.png: SWAT: [[gerrit:534151{{!}}Add high-density logos for Wikidata (T230120)]] Part I (duration: 00m 56s)
* 10:42 marostegui: Start event scheduler on db1115 [[phab:T231769|T231769]]
* 10:23 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp2002 - [[phab:T231859|T231859]]
* 10:20 marostegui: Start MySQL on db1115  without the event scheduler - [[phab:T231769|T231769]]
* 10:12 marostegui: Stop MySQL on db1115  without the event scheduler - [[phab:T231769|T231769]]
* 10:12 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp5001 - [[phab:T231859|T231859]]
* 10:11 @: helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' .
* 10:11 marostegui: Tendril/dbtree will be unavailable for a few minutes [[phab:T231769|T231769]]
* 10:11 marostegui: Stop MySQL on db1115 - [[phab:T231769|T231769]]
* 10:09 vgutierrez: uploaded trafficserver 8.0.5-1wm5 to apt.wikimedia.org (stretch) - [[phab:T231533|T231533]] [[phab:T231859|T231859]]
* 09:33 moritzm: upgrading mw servers in codfw to 7.2.22
* 09:19 _joe_: uploaded envoyproxy to buster
* 08:56 moritzm: upgrading mw1238-mw1258 to PHP 7.2.22
* 08:42 marostegui: Stop HAproxy on dbproxy1005 - [[phab:T231967|T231967]]
* 08:37 moritzm: upgrading API canaries in eqiad to 7.2.22
* 08:26 marostegui: Reboot db1135 to pick up new kernel - [[phab:T231403|T231403]]
* 07:50 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db2047 from config [[phab:T231852|T231852]] (duration: 00m 54s)
* 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db2047 from config [[phab:T231852|T231852]] (duration: 00m 57s)
* 07:21 mutante: ununpentium - a2dismod ssl - systemctl restart apache2
* 05:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
* 05:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
* 02:46 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/resources/src/startup/mediawiki.js: {{Gerrit|8a1b13026}} (duration: 00m 55s)
* 02:42 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/resources/src/mediawiki.base/mediawiki.base.js: {{Gerrit|8a1b13026}} (duration: 00m 56s)
* 02:21 chaomodus: extending downtime on netmon1002 and netmon2001, netbox1001, netbox2001, netboxdb1001 and netbox2001 should be stable but are still being debugged
* 01:02 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|ed5297c10}} / [[phab:T217830|T217830]] (duration: 00m 59s)
* 01:02 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: {{Gerrit|ed5297c10}} / [[phab:T217830|T217830]] (duration: 00m 59s)
* 00:02 chaomodus: installing and setting up netbox instances [[phab:T223291|T223291]]
* 00:02 chaomodus: installing and setting up netbox instances [[phab:T223291|T223291]]

Revision as of 00:55, 5 September 2019

2019-09-05

  • 00:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 53s)
  • 00:54 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out write of variant config into MWConfigCacheGenerator, part 1 (duration: 00m 56s)
  • 00:04 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out load of variant config into MWConfigCacheGenerator, part 2 (duration: 00m 55s)
  • 00:02 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out load of variant config into MWConfigCacheGenerator, part 1 (duration: 00m 55s)

2019-09-04

  • 23:36 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: CommonSettings: Factor out variant config generation into MWConfigCacheGenerator, part 2 (duration: 00m 55s)
  • 23:33 jforrester@deploy1001: Synchronized multiversion/MWConfigCacheGenerator.php: CommonSettings: Factor out variant config generation into MWConfigCacheGenerator, part 1 (duration: 00m 54s)
  • 23:05 urandom: decommission restbase-dev1004-b (Cassandra) -- T224554
  • 21:58 andrewbogott: attached to console on cumin1001, found it in bios 'system settings', exited, allowed boot to continue. No idea how it got there — spontaneous reboot?
  • 21:12 crusnov@deploy1001: Finished deploy [netbox/deploy@367ca84]: (no justification provided) (duration: 08m 55s)
  • 21:03 crusnov@deploy1001: Started deploy [netbox/deploy@367ca84]: (no justification provided)
  • 20:14 urandom: decommission restbase-dev1004-a (Cassandra) -- T224554
  • 20:00 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 19:35 hashar@deploy1001: rebuilt and synchronized wikiversions files: rollback wikidatawiki to 1.34.0-wmf.20 for T232035 - T220746
  • 19:33 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 19:17 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 19:00 hashar@deploy1001: Synchronized php: group1 wikis to 1.34.0-wmf.21 (duration: 00m 54s)
  • 18:59 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.21
  • 17:59 jforrester@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/modules/homepage/: T229271 Homepage: Unbreak question dialogs on mobile (duration: 00m 56s)
  • 17:47 jforrester@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T150418 Fix HTML blacklist inheritance to avoid copy-pasted read <ref>s again (duration: 00m 57s)
  • 17:45 jforrester@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T150418 Fix HTML blacklist inheritance to avoid copy-pasted read <ref>s again (duration: 00m 56s)
  • 17:43 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 - take 2 (duration: 00m 55s)
  • 17:34 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch all non-low-traffic jobs to eventgate - T228705 (duration: 00m 56s)
  • 17:32 ottomata: Switch all non-low-traffic jobs to eventgate - T228705
  • 17:14 @: helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 16:50 @: helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' .
  • 16:48 joal@deploy1001: Finished deploy [analytics/refinery@2322f10]: Fix for yesterday regular analytics deploy (duration: 53m 16s)
  • 16:40 Lucas_WMDE: Morning SWAT done
  • 16:38 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/AbuseFilter: SWAT: Fix filter validation in ViewEdit (T231985) (duration: 00m 58s)
  • 16:11 kartik@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 533172|Move ContentTranslation out of Beta in jvwiki (T231207) (duration: 00m 56s)
  • 15:55 joal@deploy1001: Started deploy [analytics/refinery@2322f10]: Fix for yesterday regular analytics deploy
  • 15:36 godog: upgrade grafana to 5.4.5 on labmon
  • 14:51 andrewbogott: reimaging cloudvirt1015 for T220853
  • 14:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove obsoleted DB config from db-eqiad.php T231642 (duration: 00m 57s)
  • 14:08 cdanis: If0dd79604 actually live on canaries now
  • 14:04 cdanis: If0dd79604 deployed to eqiad MW canaries T231642
  • 13:59 moritzm: installing nghttp2 security updates
  • 13:59 cdanis: manually testing If0dd79604 on mwdebug1001
  • 13:47 _joe_: restarting php7.2-fpm across the fleet to pick up the apc.ttl removal
  • 13:20 cdanis@deploy1001: Synchronized wmf-config/db-codfw.php: a8dc4c4a0 db-codfw: remove obsoleted DB config T231642 (duration: 00m 55s)
  • 13:20 oblivian@cumin1001: END (PASS) - Cookbook sre.mediawiki.restart-appservers (exit_code=0)
  • 13:17 oblivian@cumin1001: START - Cookbook sre.mediawiki.restart-appservers
  • 13:17 oblivian@cumin1001: END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99)
  • 13:17 oblivian@cumin1001: START - Cookbook sre.mediawiki.restart-appservers
  • 12:56 cdanis: manually testing I1bc6d1603 on mwdebug2002
  • 12:49 gehel: reset kartotherian password on maps slaves - T231964
  • 12:36 gehel: restart kartotherian on maps1001 - T231964
  • 11:52 dcausse: EU SWAT done
  • 11:49 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T231194: [cirrus] Reenable sanity checks (duration: 00m 56s)
  • 11:47 dcausse@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/CirrusSearch/: T159321: Add morelikethis a non-greedy version of the morelike keyword (duration: 00m 57s)
  • 11:47 Amir1: start of ladsgroup@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 2000000 --sleep 2 > ~/rebuildItemTerms.out 2> rebuildItemTerms.err (T225056). This is going to take a while. On screen
  • 11:38 moritzm: upgrading mw1339-mw1348 to PHP 7.2.22
  • 11:37 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m (T225055) (duration: 00m 55s)
  • 11:32 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add high-density logos for the Incubator (T230122) (duration: 00m 56s)
  • 11:28 ladsgroup@deploy1001: Synchronized static/images/project-logos/incubatorwiki-2x.png: SWAT: Add high-density logos for the Incubator (T230122) Part II (duration: 00m 54s)
  • 11:27 ladsgroup@deploy1001: Synchronized static/images/project-logos/incubatorwiki-1.5x.png: SWAT: Add high-density logos for the Incubator (T230122) Part I (duration: 00m 52s)
  • 11:24 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ printf '%s\n' 'https://en.wikipedia.org/static/images/project-logos/wikidatawiki-1.5x.png' | mwscript purgeList.php wikidatawiki # T230120
  • 11:18 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add high-density logos for Wikidata (T230120) (duration: 00m 55s)
  • 11:14 ladsgroup@deploy1001: Synchronized static/images/project-logos/wikidatawiki-2x.png: SWAT: Add high-density logos for Wikidata (T230120) Part II (duration: 00m 56s)
  • 11:12 ladsgroup@deploy1001: Synchronized static/images/project-logos/wikidatawiki-1.5x.png: SWAT: Add high-density logos for Wikidata (T230120) Part I (duration: 00m 56s)
  • 10:42 marostegui: Start event scheduler on db1115 T231769
  • 10:23 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp2002 - T231859
  • 10:20 marostegui: Start MySQL on db1115 without the event scheduler - T231769
  • 10:12 marostegui: Stop MySQL on db1115 without the event scheduler - T231769
  • 10:12 vgutierrez: upgrading ATS to 8.0.5-1wm5 on cp5001 - T231859
  • 10:11 @: helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' .
  • 10:11 marostegui: Tendril/dbtree will be unavailable for a few minutes T231769
  • 10:11 marostegui: Stop MySQL on db1115 - T231769
  • 10:09 vgutierrez: uploaded trafficserver 8.0.5-1wm5 to apt.wikimedia.org (stretch) - T231533 T231859
  • 09:33 moritzm: upgrading mw servers in codfw to 7.2.22
  • 09:19 _joe_: uploaded envoyproxy to buster
  • 08:56 moritzm: upgrading mw1238-mw1258 to PHP 7.2.22
  • 08:42 marostegui: Stop HAproxy on dbproxy1005 - T231967
  • 08:37 moritzm: upgrading API canaries in eqiad to 7.2.22
  • 08:26 marostegui: Reboot db1135 to pick up new kernel - T231403
  • 07:50 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db2047 from config T231852 (duration: 00m 54s)
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db2047 from config T231852 (duration: 00m 57s)
  • 07:21 mutante: ununpentium - a2dismod ssl - systemctl restart apache2
  • 05:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • 05:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime
  • 02:46 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/resources/src/startup/mediawiki.js: 8a1b13026 (duration: 00m 55s)
  • 02:42 krinkle@deploy1001: Synchronized php-1.34.0-wmf.21/resources/src/mediawiki.base/mediawiki.base.js: 8a1b13026 (duration: 00m 56s)
  • 02:21 chaomodus: extending downtime on netmon1002 and netmon2001, netbox1001, netbox2001, netboxdb1001 and netbox2001 should be stable but are still being debugged
  • 01:02 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s)
  • 00:02 chaomodus: installing and setting up netbox instances T223291

2019-09-03

  • 23:57 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s)
  • 23:56 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s)
  • 23:54 niharika29@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 55s)
  • 23:52 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 56s)
  • 23:42 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/tests/phpunit/: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 57s)
  • 23:41 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/includes/block: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 55s)
  • 23:28 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable and configure ORES damaging and goodfaith on zhwiki T225562 (duration: 00m 58s)
  • 23:10 ebernhardson: production-search-eqiad all indices index.merge.policy.deletes_pct_allowed=20
  • 22:54 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208694 Set CentralNotice's wgNoticeProjects for wikimedia (duration: 00m 59s)
  • 22:45 eileen: process-control config revision is 100334de4a adjust silverpop schedule
  • 19:42 XioNoX: rollback OSPF metric change on eqiad-codfw Zayo link (1320->320)
  • 19:20 fdans@deploy1001: Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided)
  • 19:18 fdans@deploy1001: Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided)
  • 19:14 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch high-traffic jobs to eventgate. Take 2 - T228705 (duration: 00m 56s)
  • 19:12 ottomata: switching jobqueue events to eventgate-main - T228705
  • 18:41 urbanecm@deploy1001: Synchronized wmf-config/: Emergency fix: GE not loading configuration properly: newbie facing feature (duration: 00m 57s)
  • 18:35 Urbanecm: Livetesting on mwdebug1002
  • 17:45 James_F: Pulled I9b64a2bb770 into wmf.21 production on the deploy server; no need to deploy to app-servers, CI-only fix.
  • 17:40 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21
  • 16:35 catrope@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/Graph/includes/ApiGraph.php: T231894 (duration: 00m 55s)
  • 16:01 joal@deploy1001: Finished deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy (duration: 136m 59s)
  • 15:55 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T227260 (duration: 00m 54s)
  • 15:32 ebernhardson: unban elastic1027 from production-search-eqiad
  • 15:07 hashar@deploy1001: rebuilt and synchronized wikiversions files: testwiki 1.34.0-wmf.21 for T231894 - T220746
  • 14:57 hashar@deploy1001: rebuilt and synchronized wikiversions files: Rollback group0 to 1.34.0-wmf.21 - T220746
  • 14:45 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21 - T220746
  • 14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Promote db1133 as wikitech master T229657 (duration: 00m 54s)
  • 14:28 hashar@deploy1001: Finished scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 (duration: 50m 09s)
  • 14:21 moritzm: upgrading app server canaries to PHP 7.2.22 T230024
  • 13:44 joal@deploy1001: Started deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy
  • 13:38 hashar@deploy1001: Started scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746
  • 13:26 hashar: Gerrit should be fine again, apparently was due to the wmf branch cut taking too much resources (sic) - T231872 filled to investigate
  • 13:25 hashar: 1.34.0-wmf.21 cut
  • 13:16 hashar: Gerrit has some random times out from time to time (no reason)
  • 13:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1073 from wikitech T229657', diff saved to https://phabricator.wikimedia.org/P9038 and previous config saved to /var/cache/conftool/dbconfig/20190903-131456-marostegui.json
  • 13:13 marostegui: Re-enable puppet on db1073 and db1133 T229657
  • 13:11 marostegui: Reload haproxy on dbproxy1005 T229657
  • 13:10 marostegui@cumin1001: dbctl commit (dc=all): 'Set wikitech back to RW after maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9037 and previous config saved to /var/cache/conftool/dbconfig/20190903-131000-marostegui.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9033 and previous config saved to /var/cache/conftool/dbconfig/20190903-130113-marostegui.json
  • 13:00 marostegui: Failover m5 from db1073 to db1133 - T229657
  • 12:52 moritzm: uploaded PHP 7.2.22 to component/php72 T230024
  • 12:39 moritzm: upgrading mwdebug2001 to PHP 7.2.22
  • 12:29 hashar: Cutting wmf/1.34.0-wmf.21 # T220746
  • 12:19 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.20
  • 12:02 marostegui: Disable puppet on db1073 and db1133 - T229657
  • 11:55 marostegui: Change topology on m5 and make everything replicate from db1133 - T229657
  • 11:48 marostegui: Downtime m5 hosts T229657
  • 11:35 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 (T225056)
  • 11:29 Amir1: EU SWAT is done
  • 11:29 Amir1: ladsgroup@mwmaint1002:~$ mwscript namespaceDupes.php bswiki --fix (T231654)
  • 11:28 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix wgMetaNamespaceTalk for bswiki (T231654) (duration: 00m 54s)
  • 11:25 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016) (duration: 00m 52s)
  • 11:11 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016) (duration: 00m 53s)
  • 11:07 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WRITE_BOTH for items term store for wikidatawiki (T225055) (duration: 00m 55s)
  • 10:17 ema: cp1083: varnish-backend-restart -- mbox lag, fetch failures
  • 09:59 _joe_: removing old lvs-related scripts from ores*
  • 09:46 moritzm: moved uid=smalyshev from cn=wmf to cn=nda
  • 09:46 mutante: install1002 - import GPG key for getenvoy repo, importing envoy for jessie with reprepro update
  • 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
  • 09:04 ema: cp1085: varnish-backend-restart, mbox lag and fetch failures
  • 09:03 gehel: reset kartotherian password -T231842
  • 08:54 ema: cp1089: varnish-backend-restart due to mbox lag and fetch failures
  • 08:49 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be
  • 08:49 ema: cp1075: pool ats-be with caching enabled T228629
  • 08:26 marostegui: Add REPLICATION grant to wikiuser and wikiadmin on db1073 with replication enabled - T229657
  • 08:21 gehel: purging maps / info.json from cache - T231842
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9031 and previous config saved to /var/cache/conftool/dbconfig/20190903-080958-marostegui.json
  • 08:04 joal@deploy1001: Finished deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try (duration: 00m 27s)
  • 08:03 joal@deploy1001: Started deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try
  • 08:02 joal@deploy1001: deploy aborted: Regular weekly analytics deploy train (duration: 27m 47s)
  • 07:16 marostegui: Change min_replicas to 6 on s1 for eqiad and codfw T231019
  • 06:39 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9029 and previous config saved to /var/cache/conftool/dbconfig/20190903-063932-marostegui.json
  • 06:10 mutante: running puppet on cp-text_eqiad to switch people.wm.org to https backend
  • 06:04 marostegui: Change min_replicas to 4 on s7 for eqiad and codfw T231019
  • 05:53 mutante: people.wikimedia.org - switching to TLS termination with envoy
  • 05:52 marostegui@cumin1001: dbctl commit (dc=all): 'Reorganize s7 codfw T230106', diff saved to https://phabricator.wikimedia.org/P9028 and previous config saved to /var/cache/conftool/dbconfig/20190903-055234-marostegui.json
  • 05:47 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Reorganize s7 codfw T230106 (duration: 00m 54s)
  • 05:22 marostegui: Rename tables on the puppet database on m1 master - T231539
  • 05:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106 (duration: 00m 54s)
  • 05:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2047 old master from s7 T230106', diff saved to https://phabricator.wikimedia.org/P9027 and previous config saved to /var/cache/conftool/dbconfig/20190903-051619-marostegui.json
  • 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db2118 to s7 codfw master (db2047 -> db2118) T230106', diff saved to https://phabricator.wikimedia.org/P9026 and previous config saved to /var/cache/conftool/dbconfig/20190903-051450-marostegui.json
  • 05:02 marostegui: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106
  • 04:50 marostegui: Drop filejournal table on s3 - T51195
  • 04:49 vgutierrez: repooling cp2002 - T231433
  • 04:36 vgutierrez: upgrading ATS to 8.0.5-1wm4 on cp2002 - T231433
  • 04:28 vgutierrez: Switching cp2002 from nginx to ats-tls - T231433

2019-09-02

  • 22:08 ebernhardson: ban elastic1027 from production-search-chi
  • 20:48 ebernhardson: restart production-search-eqiad on elastic1027 again
  • 20:33 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@453ee8a]: Make osm-pbf source private (T231842) (duration: 02m 09s)
  • 20:31 mbsantos@deploy1001: Started deploy [kartotherian/deploy@453ee8a]: Make osm-pbf source private (T231842)
  • 19:54 ebernhardson: restart elasticsearch_6@production-search-eqiad on elastic1027
  • 17:57 mateusbs17: regenerating tiles from z0 to z9 in eqiad and codfw- T231691, T230511
  • 15:08 moritzm: installing libssh2 security updates
  • 14:36 moritzm: installing ghostscript updates on thumbor1001
  • 14:24 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 14:21 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 14:10 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 13:44 akosiaris: resync the sessionstore staging release as there was wrong port mapping (port 8080 instead of 8081) for both netpol and service
  • 13:43 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 13:40 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 13:09 vgutierrez: upgrading prometheus-trafficserver-exporter to version 0.3.2 on the cache cluster - T231533
  • 12:58 vgutierrez: upgrading prometheus-trafficserver-exporter to version 0.3.2 on cp5001 - T231533
  • 12:46 vgutierrez: uploaded prometheus-trafficserver-exporter 0.3.2 to apt.wikimedia.org (stretch) - T231533
  • 12:40 moritzm: installing freetype security updates on jessie (stretch/buster already fixed)
  • 11:23 moritzm: installing apache2 security updates on jessie
  • 11:18 moritzm: imported apache2 2.4.10-10+deb8u15+wmf1 to apt.wikimedia.org/jessie-wikimedia (rebuild of latest Jessie update against our patches)
  • 10:25 moritzm: installing libav security updates
  • 10:07 moritzm: installing subversion security updates on jessie
  • 09:21 marostegui: Drop filejournal table on s7 - T51195
  • 09:15 marostegui: Drop filejournal table on s1 - T51195
  • 08:45 marostegui: Drop filejournal table on s8 - T51195
  • 08:27 marostegui: Drop filejournal table on labtestwiki - T51195
  • 08:25 marostegui: Drop filejournal table on s2 - T51195
  • 08:15 godog: upgrade grafana to 5.4.5 on grafana1001
  • 08:12 godog: update amd-rocm debian repository gpg key (same id, new expiration)
  • 07:34 marostegui: Drop filejournal table on s4 - T51195
  • 07:26 marostegui: Drop filejournal table on s5 - T51195
  • 07:17 marostegui: Drop filejournal table on s6 - T51195
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db2046 from config T231767 (duration: 00m 53s)
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db2046 from config T231767 (duration: 00m 55s)

2019-09-01

  • 17:53 Urbanecm: Run mwscript extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --wiki=enwikiquote --verbose (T231137)
  • 17:45 Urbanecm: Run mwscript extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --wiki=metawiki --verbose (T231137)
  • 17:33 Urbanecm: Run foreachwikiindblist group1.dblist extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --dry-run --verbose (T231137)
  • 17:29 Urbanecm: Previous should be *group0.dblist (T231137)
  • 17:29 Urbanecm: Run foreachwikiindblist group0 extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --dry-run --verbose (T231137)


Archives

See Server admin log/Archives.