You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log

From Wikitech-static
Revision as of 01:02, 4 September 2019 by imported>Stashbot (krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s))
Jump to navigation Jump to search

2019-09-04

  • 01:02 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s)
  • 00:02 chaomodus: installing and setting up netbox instances T223291

2019-09-03

  • 23:57 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s)
  • 23:56 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s)
  • 23:54 niharika29@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 55s)
  • 23:52 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 56s)
  • 23:42 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/tests/phpunit/: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 57s)
  • 23:41 niharika29@deploy1001: Synchronized php-1.34.0-wmf.20/includes/block: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 55s)
  • 23:28 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable and configure ORES damaging and goodfaith on zhwiki T225562 (duration: 00m 58s)
  • 23:10 ebernhardson: production-search-eqiad all indices index.merge.policy.deletes_pct_allowed=20
  • 22:54 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208694 Set CentralNotice's wgNoticeProjects for wikimedia (duration: 00m 59s)
  • 22:45 eileen: process-control config revision is 100334de4a adjust silverpop schedule
  • 19:42 XioNoX: rollback OSPF metric change on eqiad-codfw Zayo link (1320->320)
  • 19:20 fdans@deploy1001: Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided)
  • 19:18 fdans@deploy1001: Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided)
  • 19:14 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch high-traffic jobs to eventgate. Take 2 - T228705 (duration: 00m 56s)
  • 19:12 ottomata: switching jobqueue events to eventgate-main - T228705
  • 18:41 urbanecm@deploy1001: Synchronized wmf-config/: Emergency fix: GE not loading configuration properly: newbie facing feature (duration: 00m 57s)
  • 18:35 Urbanecm: Livetesting on mwdebug1002
  • 17:45 James_F: Pulled I9b64a2bb770 into wmf.21 production on the deploy server; no need to deploy to app-servers, CI-only fix.
  • 17:40 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21
  • 16:35 catrope@deploy1001: Synchronized php-1.34.0-wmf.21/extensions/Graph/includes/ApiGraph.php: T231894 (duration: 00m 55s)
  • 16:01 joal@deploy1001: Finished deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy (duration: 136m 59s)
  • 15:55 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T227260 (duration: 00m 54s)
  • 15:32 ebernhardson: unban elastic1027 from production-search-eqiad
  • 15:07 hashar@deploy1001: rebuilt and synchronized wikiversions files: testwiki 1.34.0-wmf.21 for T231894 - T220746
  • 14:57 hashar@deploy1001: rebuilt and synchronized wikiversions files: Rollback group0 to 1.34.0-wmf.21 - T220746
  • 14:45 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21 - T220746
  • 14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Promote db1133 as wikitech master T229657 (duration: 00m 54s)
  • 14:28 hashar@deploy1001: Finished scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 (duration: 50m 09s)
  • 14:21 moritzm: upgrading app server canaries to PHP 7.2.22 T230024
  • 13:44 joal@deploy1001: Started deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy
  • 13:38 hashar@deploy1001: Started scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746
  • 13:26 hashar: Gerrit should be fine again, apparently was due to the wmf branch cut taking too much resources (sic) - T231872 filled to investigate
  • 13:25 hashar: 1.34.0-wmf.21 cut
  • 13:16 hashar: Gerrit has some random times out from time to time (no reason)
  • 13:14 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1073 from wikitech T229657', diff saved to https://phabricator.wikimedia.org/P9038 and previous config saved to /var/cache/conftool/dbconfig/20190903-131456-marostegui.json
  • 13:13 marostegui: Re-enable puppet on db1073 and db1133 T229657
  • 13:11 marostegui: Reload haproxy on dbproxy1005 T229657
  • 13:10 marostegui@cumin1001: dbctl commit (dc=all): 'Set wikitech back to RW after maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9037 and previous config saved to /var/cache/conftool/dbconfig/20190903-131000-marostegui.json
  • 13:01 marostegui@cumin1001: dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9033 and previous config saved to /var/cache/conftool/dbconfig/20190903-130113-marostegui.json
  • 13:00 marostegui: Failover m5 from db1073 to db1133 - T229657
  • 12:52 moritzm: uploaded PHP 7.2.22 to component/php72 T230024
  • 12:39 moritzm: upgrading mwdebug2001 to PHP 7.2.22
  • 12:29 hashar: Cutting wmf/1.34.0-wmf.21 # T220746
  • 12:19 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.20
  • 12:02 marostegui: Disable puppet on db1073 and db1133 - T229657
  • 11:55 marostegui: Change topology on m5 and make everything replicate from db1133 - T229657
  • 11:48 marostegui: Downtime m5 hosts T229657
  • 11:35 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --to-id 1000 --sleep 2 (T225056)
  • 11:29 Amir1: EU SWAT is done
  • 11:29 Amir1: ladsgroup@mwmaint1002:~$ mwscript namespaceDupes.php bswiki --fix (T231654)
  • 11:28 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix wgMetaNamespaceTalk for bswiki (T231654) (duration: 00m 54s)
  • 11:25 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016) (duration: 00m 52s)
  • 11:11 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016) (duration: 00m 53s)
  • 11:07 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WRITE_BOTH for items term store for wikidatawiki (T225055) (duration: 00m 55s)
  • 10:17 ema: cp1083: varnish-backend-restart -- mbox lag, fetch failures
  • 09:59 _joe_: removing old lvs-related scripts from ores*
  • 09:46 moritzm: moved uid=smalyshev from cn=wmf to cn=nda
  • 09:46 mutante: install1002 - import GPG key for getenvoy repo, importing envoy for jessie with reprepro update
  • 09:16 hashar: Deploy refactor of Zuul pipelines which might mean that some repos/branches would miss jobs or have extra unwanted jobs. In such case please fill in a task against #continuous-integration-config
  • 09:04 ema: cp1085: varnish-backend-restart, mbox lag and fetch failures
  • 09:03 gehel: reset kartotherian password -T231842
  • 08:54 ema: cp1089: varnish-backend-restart due to mbox lag and fetch failures
  • 08:49 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be
  • 08:49 ema: cp1075: pool ats-be with caching enabled T228629
  • 08:26 marostegui: Add REPLICATION grant to wikiuser and wikiadmin on db1073 with replication enabled - T229657
  • 08:21 gehel: purging maps / info.json from cache - T231842
  • 08:10 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9031 and previous config saved to /var/cache/conftool/dbconfig/20190903-080958-marostegui.json
  • 08:04 joal@deploy1001: Finished deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try (duration: 00m 27s)
  • 08:03 joal@deploy1001: Started deploy [analytics/refinery@4810dfa]: Regular weekly analytics deploy train - Second try
  • 08:02 joal@deploy1001: deploy aborted: Regular weekly analytics deploy train (duration: 27m 47s)
  • 07:16 marostegui: Change min_replicas to 6 on s1 for eqiad and codfw T231019
  • 06:39 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1133 with weight 0 T229657', diff saved to https://phabricator.wikimedia.org/P9029 and previous config saved to /var/cache/conftool/dbconfig/20190903-063932-marostegui.json
  • 06:10 mutante: running puppet on cp-text_eqiad to switch people.wm.org to https backend
  • 06:04 marostegui: Change min_replicas to 4 on s7 for eqiad and codfw T231019
  • 05:53 mutante: people.wikimedia.org - switching to TLS termination with envoy
  • 05:52 marostegui@cumin1001: dbctl commit (dc=all): 'Reorganize s7 codfw T230106', diff saved to https://phabricator.wikimedia.org/P9028 and previous config saved to /var/cache/conftool/dbconfig/20190903-055234-marostegui.json
  • 05:47 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Reorganize s7 codfw T230106 (duration: 00m 54s)
  • 05:22 marostegui: Rename tables on the puppet database on m1 master - T231539
  • 05:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106 (duration: 00m 54s)
  • 05:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2047 old master from s7 T230106', diff saved to https://phabricator.wikimedia.org/P9027 and previous config saved to /var/cache/conftool/dbconfig/20190903-051619-marostegui.json
  • 05:14 marostegui@cumin1001: dbctl commit (dc=all): 'Promote db2118 to s7 codfw master (db2047 -> db2118) T230106', diff saved to https://phabricator.wikimedia.org/P9026 and previous config saved to /var/cache/conftool/dbconfig/20190903-051450-marostegui.json
  • 05:02 marostegui: Promote db2118 to s7 codfw master (db2047 -> db2118) T230106
  • 04:50 marostegui: Drop filejournal table on s3 - T51195
  • 04:49 vgutierrez: repooling cp2002 - T231433
  • 04:36 vgutierrez: upgrading ATS to 8.0.5-1wm4 on cp2002 - T231433
  • 04:28 vgutierrez: Switching cp2002 from nginx to ats-tls - T231433

2019-09-02

  • 22:08 ebernhardson: ban elastic1027 from production-search-chi
  • 20:48 ebernhardson: restart production-search-eqiad on elastic1027 again
  • 20:33 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@453ee8a]: Make osm-pbf source private (T231842) (duration: 02m 09s)
  • 20:31 mbsantos@deploy1001: Started deploy [kartotherian/deploy@453ee8a]: Make osm-pbf source private (T231842)
  • 19:54 ebernhardson: restart elasticsearch_6@production-search-eqiad on elastic1027
  • 17:57 mateusbs17: regenerating tiles from z0 to z9 in eqiad and codfw- T231691, T230511
  • 15:08 moritzm: installing libssh2 security updates
  • 14:36 moritzm: installing ghostscript updates on thumbor1001
  • 14:24 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 14:21 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 14:10 @: helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' .
  • 13:44 akosiaris: resync the sessionstore staging release as there was wrong port mapping (port 8080 instead of 8081) for both netpol and service
  • 13:43 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 13:40 @: helmfile [STAGING] Ran 'sync' command on namespace 'sessionstore' for release 'staging' .
  • 13:09 vgutierrez: upgrading prometheus-trafficserver-exporter to version 0.3.2 on the cache cluster - T231533
  • 12:58 vgutierrez: upgrading prometheus-trafficserver-exporter to version 0.3.2 on cp5001 - T231533
  • 12:46 vgutierrez: uploaded prometheus-trafficserver-exporter 0.3.2 to apt.wikimedia.org (stretch) - T231533
  • 12:40 moritzm: installing freetype security updates on jessie (stretch/buster already fixed)
  • 11:23 moritzm: installing apache2 security updates on jessie
  • 11:18 moritzm: imported apache2 2.4.10-10+deb8u15+wmf1 to apt.wikimedia.org/jessie-wikimedia (rebuild of latest Jessie update against our patches)
  • 10:25 moritzm: installing libav security updates
  • 10:07 moritzm: installing subversion security updates on jessie
  • 09:21 marostegui: Drop filejournal table on s7 - T51195
  • 09:15 marostegui: Drop filejournal table on s1 - T51195
  • 08:45 marostegui: Drop filejournal table on s8 - T51195
  • 08:27 marostegui: Drop filejournal table on labtestwiki - T51195
  • 08:25 marostegui: Drop filejournal table on s2 - T51195
  • 08:15 godog: upgrade grafana to 5.4.5 on grafana1001
  • 08:12 godog: update amd-rocm debian repository gpg key (same id, new expiration)
  • 07:34 marostegui: Drop filejournal table on s4 - T51195
  • 07:26 marostegui: Drop filejournal table on s5 - T51195
  • 07:17 marostegui: Drop filejournal table on s6 - T51195
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db2046 from config T231767 (duration: 00m 53s)
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db2046 from config T231767 (duration: 00m 55s)

2019-09-01

  • 17:53 Urbanecm: Run mwscript extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --wiki=enwikiquote --verbose (T231137)
  • 17:45 Urbanecm: Run mwscript extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --wiki=metawiki --verbose (T231137)
  • 17:33 Urbanecm: Run foreachwikiindblist group1.dblist extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --dry-run --verbose (T231137)
  • 17:29 Urbanecm: Previous should be *group0.dblist (T231137)
  • 17:29 Urbanecm: Run foreachwikiindblist group0 extensions/AbuseFilter/maintenance/fixFirstBlockautopromoteEntries.php --dry-run --verbose (T231137)


Archives

See Server admin log/Archives.