You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log

From Wikitech-static
Revision as of 00:26, 6 November 2018 by imported>Stashbot (maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s))
Jump to navigation Jump to search

2018-11-06

2018-11-05

  • 22:41 mutante: sodium - reboot after disk replacement (T202705)
  • 21:51 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100 (duration: 01m 02s)
  • 21:50 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100
  • 21:49 arlolra: Updated Parsoid to 8ed698b (T205334, T208360)
  • 21:42 mobrovac@deploy1001: Started restart [zotero/translation-server@50f216a]: Free up some memory
  • 21:41 arlolra@deploy1001: Finished deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b (duration: 10m 59s)
  • 21:38 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Removed decomissioned citoid url T133001 (duration: 00m 53s)
  • 21:30 arlolra@deploy1001: Started deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b
  • 21:23 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2 (duration: 09m 18s)
  • 21:14 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2
  • 21:10 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 (duration: 12m 15s)
  • 21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Ensure all wikis on 1.33.0-wmf.2
  • 20:58 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324
  • 20:38 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table (duration: 03m 40s)
  • 20:35 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table
  • 19:53 akosiaris: do a depool, scap deploy, scap wikiversions-compile, hhvm restart and then a pool in eqiad mediawiki servers
  • {{safesubst:SAL entry|1=19:50 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:470868|Disable page issues A/B test (duration: 00m 53s)}}
  • 19:44 sbisson@deploy1001: Synchronized php-1.33.0-wmf.2/includes/block/BlockRestriction.php: SWAT: BlockRestriction::update() unnecessarily does a SELECT on the page table. (duration: 01m 00s)
  • 19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable error logging for WikimediaEvents (duration: 00m 52s)
  • 19:12 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GrowthExperiments logging channel (duration: 00m 53s)
  • 18:57 _joe_: restarting hhvm on mwdebug1002
  • 18:44 godog: pool graphite1004 for reads - T196484
  • 18:43 XioNoX: delete asw2-b - asw-b interface - T183585
  • 18:41 XioNoX: remove asw-b-eqiad from LibreNMS - T183585
  • 18:37 XioNoX: remove vrrp priority 70 on cr2-eqiad:ae2 to failback VIPs to cr2 - T183585
  • 18:26 XioNoX: re-enable ae2 on cr2-eqiad - T183585
  • 18:21 thcipriani: rollback mwdebug1001 group2 wikis
  • 18:13 thcipriani: testing php-1.33.0-wmf.2 on group2 wikis on mwdebug1001
  • 18:05 XioNoX: disable ae2 on cr2-eqiad - T183585
  • 18:02 XioNoX: set vrrp priority 70 on cr2-eqiad:ae2 to failover VIP to cr1 - T183585
  • 16:49 XioNoX: Update LLDP config on cr3-ulsfo - T208630
  • 16:48 vgutierrez: uploaded certcentral 0.5 to apt.wikimedia.org (stretch) - T208572 T208378
  • 16:06 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR to read-new on all wikis (T198308) (duration: 00m 55s)
  • 13:57 jynus_: increase consistency of db2050, dbstore2002 s3 after them catching up replication T208462
  • 12:33 ladsgroup@deploy1001: Finished deploy [ores/deploy@096ffb3]: T208577 T181632 T208608 (duration: 22m 58s)
  • 12:23 zeljkof: EU SWAT finished
  • 12:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase wikidata dispatchers to 3 (duration: 00m 54s)
  • 12:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgForeignUploadTargets to [] for zhwiki (T208397) (duration: 00m 54s)
  • 12:10 ladsgroup@deploy1001: Started deploy [ores/deploy@096ffb3]: T208577 T181632 T208608
  • 12:05 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Anniversary logo for cswiki" (T207589) (duration: 00m 58s)
  • 10:02 godog: reformat xfs filesystems on ms-be1040 - T199198
  • 09:17 elukey@deploy1001: Finished deploy [analytics/refinery@9d39efa]: fixing stat1004 (duration: 00m 04s)
  • 09:17 elukey@deploy1001: Started deploy [analytics/refinery@9d39efa]: fixing stat1004
  • 09:08 joal@deploy1001: Finished deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy (duration: 05m 21s)
  • 09:02 joal@deploy1001: Started deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy

2018-11-04

  • 23:42 jynus_: deleting the same row on all s8 broken servers
  • 23:39 jynus_: deleting one row on db1104
  • 20:38 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: T176232 - Ia43626584e (duration: 01m 17s)
  • 18:32 jynus_: reduce temp. consistency level of s4, s5, and s6 codfw masters to prevent excessive lagging due to ongoing mediawiki core maintenance
  • 08:42 eileen: process-control config revision is e832b5a04a renable running job list (all jobs on again now0
  • 08:38 eileen: process-control config revision is e16b2c1c61 renable jobs
  • 02:00 eileen: I think I got the rest of the jobs off process-control config revision is 4422254128
  • 01:52 eileen: process-control config revision is 6ec67b3d01 - also turn off omnirecipient repair job
  • 01:40 eileen: process-control config revision is 5b72cfe874 - reapply turn off q jobs

2018-11-03

  • 09:35 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2018-11-02

  • 17:04 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}
  • 16:54 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1002
  • 16:43 _joe_: live-hacking removal of time limit on mwdebug1001
  • 16:32 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1001
  • 15:12 jynus: restarting replication @ db2074 after db2094:s3 table fix T208565
  • 15:00 jynus: stopping replication on db2074 to fix db2094:s3 T208565
  • 14:01 vgutierrez: reimaging eeden.wikimedia.org as jessie test system - T208583
  • 11:43 jynus: ignoring cawikimedia.archive replication on db2094:s3 until a reimport happens T208565
  • 11:29 jijiki: Rebooting mw2244 (spare system) for maintenance
  • 10:52 ema: restart varnish-be on cp3032 T208574
  • 08:19 jynus: performing alter table on dbstore2002 s3 and reducing consistency to improve recovery time T208462 T204006
  • 08:01 jynus: reducing consitency on db2050 to improve recovery time T208462
  • 07:59 jynus: performing alter table on db2050 T208462 T204006
  • 07:38 godog: reformat ms-be1043 xfs filesystems - T199198
  • 07:38 jynus: reducing consistency temporarily (flush, binlog sync) at db2040 to prevent lagging
  • 07:26 jynus: reducing consistency temporarily (flush, binlog sync) at db2035 to prevent lagging

2018-11-01

  • 23:01 shdubsh: restart hhvm on mw1261
  • 22:29 ejegg: restarted fundraising queue consumer jobs
  • 22:21 ejegg: updated fundraising CiviCRM from 65130ef3dd to 042eeaeca9
  • 22:18 ejegg: turned off fundraising queue jobs for civi update
  • 22:12 _joe_: rolling restart of hhvm on appservers and api in eqiad
  • 22:09 shdubsh: cumin -b 2 -s 30 "O:mediawiki::appserver and *.eqiad.wmnet" "restart-hhvm"
  • 22:05 _joe_: restarting hhvm on mw1238,1240
  • 22:02 _joe_: restart hhvm on mw1244
  • 21:52 shdubsh: restart hhvm on mw1247
  • 21:49 _joe_: depooling mw1238 for debugging
  • 21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group2 back to 1.33.0-wmf.1
  • 20:55 hoo: Restarted hhvm on mwdebug2002
  • 19:40 hoo: Ran "UPDATE wb_changes_dispatch SET chd_seen = '775203911' WHERE chd_site LIKE '%wikt%' AND chd_seen < '775180000';" on wikidata master (dispatching for wiktionaries)
  • 19:00 hoo@deploy1001: Synchronized php-1.33.0-wmf.1/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 01m 01s)
  • 18:38 hoo@deploy1001: Synchronized php-1.33.0-wmf.2/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 00m 55s)
  • 18:25 jijiki: Enabling puppet on mw servers (T206923)
  • 18:19 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove now redundant Wikidata config for wiktionary (T208317) (duration: 00m 54s)
  • 18:12 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Add all wiktionaries to wikidataclient.dblist, sort list (T208317) (duration: 00m 57s)
  • 18:02 gehel: restart nginx on relforge100*
  • 17:57 jijiki: Disabling puppet on mw servers (T206923)
  • 16:07 anomie@mwmaint1002: Running migrateComments.php on section 4 wikis for T166733
  • 13:46 anomie@mwmaint1002: Running migrateComments.php on remaining section 3 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 7 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on wikitech for T166733
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on wikitech for T188132
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 6 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 8 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 8 wikis for T188132
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 7 wikis for T188132
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 6 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 5 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 1 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 2 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 5 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 4 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on remaining section 3 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 2 wikis for T188132
  • 13:35 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 1 wikis for T188132
  • 12:50 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: List wikidataclient-test in CS.php dblists T208488 (duration: 00m 57s)
  • 09:10 elukey: added a tmux session on mw1314m mw1344, mw1316 that checks mcrouter stats every 10s
  • 00:58 onimisionipe: repooling wdqs1004. It has caught up on lag with others
  • 00:22 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/extension.json: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (invalidate RL cache) (duration: 00m 53s)
  • 00:20 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightWindow.js: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (duration: 00m 54s)
  • 00:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates (duration: 03m 00s)
  • 00:10 mobrovac@deploy1001: Started deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates
  • 00:07 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)
  • 00:05 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)


Archives

See Server admin log/Archives.