You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(twentyafterfour@deploy1001: Synchronized wmf-config/: SWAT: tested on mwdebug1001, now syncing wmf-config settings to the whole cluster (duration: 00m 59s))
imported>Stashbot
(jynus: setup test s3 replication channel on db1110 (filtered))
Line 1: Line 1:
== 2018-09-18 ==
== 2018-10-02 ==
* 23:21 twentyafterfour@deploy1001: Synchronized wmf-config/: SWAT: tested on mwdebug1001, now syncing wmf-config settings to the whole cluster (duration: 00m 59s)
* 16:53 jynus: setup test s3 replication channel on db1110 (filtered)
* 23:15 twentyafterfour: SWAT: deploying 3 patches for ebernhardson: {{Gerrit|77deae12f}} {{Gerrit|7009fe473}} and {{Gerrit|12cc420fc}}
* 16:49 XioNoX: assign 10.195.0.129/29 to pfw3-codfw:reth0.2133 - [[phab:T204271|T204271]]
* 22:10 bblack: depooled ulsfo, some uknown router issue
* 16:38 cmjohnson1: swapping failed disk db1067 [[phab:T205780|T205780]]
* 21:42 bblack: authdns1001 seems stable/fine running gdnsd-2.99.9-beta so far.  If issues crop up later, don't hesitate to (a) downgrade back to stretch-backports gdnsd-2.3.0-1~bpo9+1 or (b) call me!
* 16:04 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@093551f]: Increase cirrusSearchLinksUpdate concurrency (duration: 01m 06s)
* 21:25 bblack: authdns1001: testing gdnsd version update (2.99.9-beta)
* 16:03 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@093551f]: Increase cirrusSearchLinksUpdate concurrency
* 20:58 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set ActorTableSchemaMigrationStage back to OLD on test wikis, mediawikiwiki ([[phab:T188327|T188327]], [[phab:T204669|T204669]]) (duration: 00m 57s)
* 15:50 marxarelli: cutting 1.32.0-wmf.24 branch
* 20:57 catrope@deploy1001: sync-file aborted: Set ActorTableSchemaMigrationStage back to OLD on test wikis, mediawikiwiki (duration: 00m 10s)
* 15:33 gehel: cleanup old cronjob (cleanup GC logs) on all elasticsearch servers
* 20:05 hashar@deploy1001: Synchronized php-1.32.0-wmf.22/extensions/OAuth/frontend/specialpages/SpecialMWOAuthListConsumers.php: Fix escapeForHtml method name - [[phab:T204757|T204757]] (duration: 00m 58s)
* 15:24 akosiaris: upgrade mathoid chart version to 0.0.11
* 19:56 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@58f9ed3]: Log errors for HTTP error [[phab:T203929|T203929]] (duration: 00m 49s)
* 15:24 akosiaris@deploy1001: scap-helm mathoid finished
* 19:55 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@58f9ed3]: Log errors for HTTP error [[phab:T203929|T203929]]
* 15:23 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
* 19:43 bblack: uploaded gdnsd-2.99.9-beta1-1+wmf1 to reprepro for stretch-wikimedia
* 15:23 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
* 19:20 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.22
* 15:23 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
* 19:18 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f] (dev-cluster): Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out (duration: 06m 32s)
* 15:21 akosiaris@deploy1001: scap-helm mathoid finished
* 19:11 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f] (dev-cluster): Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out
* 15:21 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
* 18:49 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out (duration: 06m 14s)
* 15:21 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
* 18:43 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out
* 15:21 akosiaris@deploy1001: scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw]
* 18:40 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out (duration: 07m 58s)
* 14:11 banyek: powering off dbstore2002.codfw.wmnet for BBU change ([[phab:T205257|T205257]])
* 18:32 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints, take 2, feed timing out
* 13:47 marostegui: Deploy schema change on s4 eqiad, this will generate lag on eqiad - [[phab:T205913|T205913]]
* 18:31 ppchelko@deploy1001: Finished deploy [restbase/deploy@4c3128f]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints (duration: 08m 55s)
* 13:06 marostegui: Deploy schema change on s7 eqiad, this will generate lag on eqiad - [[phab:T205913|T205913]]
* 18:22 ppchelko@deploy1001: Started deploy [restbase/deploy@4c3128f]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements, don't monitor PDF endpoints
* 12:47 banyek: converting enwiki.content to TokuDB on host dbstrore1002 ([[phab:T205544|T205544]])
* 18:08 ppchelko@deploy1001: Finished deploy [restbase/deploy@55100d4]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements (duration: 03m 37s)
* 12:47 banyek: converting enwiki.contents to TokuDB on host dbstrore1002 ([[phab:T205544|T205544]])
* 18:04 ppchelko@deploy1001: Started deploy [restbase/deploy@55100d4]: Remove restrictions table [[phab:T203835|T203835]], Split 25% PDF traffic to Proton [[phab:T186748|T186748]], Metrics endpoints improvements
* 11:58 banyek: converting wikidatawiki.slots to TokuDB on host dbstrore1002 ([[phab:T205544|T205544]])
* 17:29 mutante: scandium - move from role(spare) to role(parsoid_testing), making it equal to ruthenium ([[phab:T201366|T201366]])
* 11:41 arturo: downtime labstore1007 load check in icinga for 1d
* 16:45 addshore@deploy1001: Synchronized php-1.32.0-wmf.22/includes/watcheditem/WatchedItemStore.php: [[gerrit:461155{{!}}WatchedItemStore::countVisitingWatchersMultiple() fix]] [[phab:T204729|T204729]] (duration: 00m 57s)
* 11:21 zeljkof: EU SWAT finished
* 16:43 addshore@deploy1001: Synchronized php-1.32.0-wmf.20/includes/watcheditem/WatchedItemStore.php: [[gerrit:461154{{!}}WatchedItemStore::countVisitingWatchersMultiple() fix]] [[phab:T204729|T204729]] (duration: 00m 59s)
* 11:19 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: SWAT: [[gerrit:463917{{!}}Use proper index on change_tag table (T205904)]] (duration: 00m 57s)
* 16:34 XioNoX: delete `filter common-infrastructure4` on cr1/2-eqiad, unused/obsolete after [[phab:T198623|T198623]]
* 10:58 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: Delay job execution while in read-only mode - [[phab:T204154|T204154]] (duration: 00m 57s)
* 16:33 mutante: mwmaint2001 - nagios-nrpe-server had status 'failed' and caused all NRPE Icinga checks to fail but recovered after simply starting it again
* 10:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2092 (duration: 00m 56s)
* 16:32 mutante: radon - re-enabled disabled puppet without reason (decom) [[phab:T202040|T202040]]
* 10:24 marostegui: Deploy schema change on db2092 - [[phab:T203709|T203709]]
* 16:32 mutante: mwmaint2001 - starting nagios-nrpe-server
* 10:24 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2092 (duration: 00m 56s)
* 16:22 XioNoX: delete `term cumin` from cr1/2-eqiad analytics filter (already permited by established-tcp term)
* 09:30 marostegui: Deploy schema change on s2 eqiad master, lag will be generated [[phab:T205913|T205913]]
* 16:02 moritzm: installing policykit-1 security updates on jessie
* 08:43 banyek: disabling puppet on es2001 and disabling backups too
* 15:55 elukey@deploy1001: Finished deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite (duration: 09m 32s)
* 08:28 marostegui: Deploy schema change on s6 eqiad master, lag will be generated [[phab:T205913|T205913]]
* 15:53 XioNoX: add cumin2001 to mr* security policies - [[phab:T204730|T204730]]
* 08:16 jynus: test recover some s3 wiki data onto db1110 (s5)
* 15:47 moritzm: installing spice security updates
* 08:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 56s)
* 15:46 elukey@deploy1001: Started deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite
* 08:04 marostegui: Deploy schema change on s5 eqiad master, lag will be generated [[phab:T205913|T205913]]
* 15:26 XioNoX: add cumin2001 to labs-in4 on cr1/2-eqiad
* 08:01 banyek: converting wikidatawiki.content to TokuDB on host dbstrore1002 ([[phab:T205544|T205544]])
* 15:23 hashar@deploy1001: Finished scap: Sync again testwiki to php-1.32.0-wmf.20, I might have screwed up l10ncache (duration: 06m 31s)
* 07:54 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 55s)
* 15:20 hashar: hashar scap for testwiki is actually php-1.32.0-wmf.22
* 07:50 marostegui: Deploy schema change on db2071 [[phab:T205913|T205913]]
* 15:19 mepps: updated process-control to {{Gerrit|c03857f6a6}}
* 07:50 mholloway-shell@deploy1001: Finished deploy [tilerator/deploy@6c80537] (maps1004): Disable event logging requests and remove HTTP proxy (duration: 00m 17s)
* 15:17 hashar@deploy1001: Started scap: Sync again testwiki to php-1.32.0-wmf.20, I might have screwed up l10ncache
* 07:49 mholloway-shell@deploy1001: Started deploy [tilerator/deploy@6c80537] (maps1004): Disable event logging requests and remove HTTP proxy
* 15:15 mepps: updated process-control to {{Gerrit|f9ab984c5d}}
* 07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 56s)
* 15:07 hashar@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.20 (duration: 35m 46s)
* 07:48 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@0bf513a] (maps1004): Remove HTTP proxy (duration: 00m 16s)
* 14:31 hashar@deploy1001: Started scap: testwiki to php-1.32.0-wmf.20
* 07:48 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@0bf513a] (maps1004): Remove HTTP proxy
* 14:19 marostegui: Drop [[phab:T153638|T153638]]_echo_XXX tables from db1123
* 07:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 56s)
* 14:02 marostegui: Drop [[phab:T153638|T153638]]_echo_XXX tables from db1095:3313 - [[phab:T153638|T153638]]
* 07:36 marostegui: Deploy schema change on db2088:3311 [[phab:T205913|T205913]]
* 13:58 papaul: upgrading BIOS on ms-be2030
* 07:36 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 55s)
* 13:57 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T203565|T203565]]: Repool db1114 and db1119 (duration: 00m 49s)
* 07:32 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
* 13:56 godog: poweroff ms-be2030 - [[phab:T204567|T204567]]
* 07:18 marostegui: Deploy schema change on db2072 [[phab:T205913|T205913]]
* 13:48 moritzm: updating intel-microcode on Debian jessie/stretch to 3.20180807a.1
* 07:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 01m 02s)
* 13:43 moritzm: installing redis security updates on maps* servers
* 05:22 _joe_: stopped tilerator on maps1004, was spamming like crazy
* 13:32 godog: repair sdd on ms-be2041 - [[phab:T199198|T199198]]
* 01:18 ejegg: updated CiviCRM from {{Gerrit|e7a620a00c}} to {{Gerrit|c353eba283}}
* 13:31 marostegui: Rename echo_XXX tables on db1095 and db1123
* 13:26 moritzm: reimaging mw2245 (spare host) to test reimages from cumin2001
* 13:24 marostegui: Drop [[phab:T153638|T153638]]_echo_XXX tables from db1078 - [[phab:T153638|T153638]]
* 13:21 hashar: Cutting branches wmf/1.32.0-wmf.22 {{!}} [[phab:T191068|T191068]]
* 13:10 marostegui: Drop [[phab:T153638|T153638]]_echo_XXX tables from db1077 - [[phab:T153638|T153638]]
* 12:12 moritzm: rebooting labnodepool1001 for kernel security update
* 10:36 jynus: stopping dbstore2002:s2 mariadb instance
* 10:31 jynus: stop db2041 to clone to dbstore2002
* 10:25 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2041 (duration: 00m 49s)
* 10:24 marostegui: Rename echo_XXX tables on db1077 and db1078  - [[phab:T153638|T153638]]
* 10:12 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Tune the s3 and s5 database loads (duration: 00m 49s)
* 10:09 marostegui: Drop [[phab:T153638|T153638]]_echo_XXX tables from dbstore2002:3313 - [[phab:T153638|T153638]]
* 10:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 00m 49s)
* 10:04 hashar: ci: updating Quibble Jenkins jobs to 0.0.26
* 10:01 marostegui: Rename echo_XXX tables on dbstore2002:3313  - [[phab:T153638|T153638]]
* 09:37 marostegui: Drop [[phab:T153638|T153638]]_echo_XXX tables from db2057 - [[phab:T153638|T153638]]
* 09:10 jynus: stop mariadb (both instances) and restart db1105 for upgrade
* 08:54 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1105 (duration: 00m 49s)
* 08:45 banyek: stop replication & stop mysql on db1119 (preparing to clone db1114)
* 08:02 godog: bounce rsyslog on wezen/lithium, tls listener was timing out in icinga
* 08:02 jynus: stop and restart db1105 for upgrade
* 07:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105 (duration: 00m 50s)
* 06:55 moritzm: installin zsh security updates
* 06:21 marostegui: Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on dbstore2001, db2079, db2082,db2083 - [[phab:T202764|T202764]]
* 06:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1013 after kernel and mariadb upgrade (duration: 00m 49s)
* 05:56 marostegui: Stop MySQL on es1013 to upgrade mariadb & kernel
* 05:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1013 for kernel and mariadb upgrade (duration: 00m 49s)
* 05:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1012 after kernel and mariadb upgrade (duration: 00m 50s)
* 05:35 marostegui: Stop MySQL on es1012 to upgrade mariadb & kernel
* 05:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1012 for kernel and mariadb upgrade (duration: 00m 49s)
* 05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2081 - [[phab:T202764|T202764]] (duration: 00m 49s)
* 05:22 marostegui: Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on db2081 - [[phab:T202764|T202764]]
* 05:22 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2081 - [[phab:T202764|T202764]] (duration: 00m 49s)
* 05:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2080 - [[phab:T202764|T202764]] (duration: 00m 49s)
* 05:08 marostegui: Drop tmp_2 and tmp_3 index from wikidatawiki.recentchanges on db2080 - [[phab:T202764|T202764]]
* 05:08 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2080 - [[phab:T202764|T202764]] (duration: 00m 51s)
* 03:01 ejegg: restarted fundraising CiviCRM jobs
* 02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Sep 18 02:46:09 UTC 2018 (duration 10m 49s)
* 02:43 eileen_: civicrm revision changed from {{Gerrit|b0d7df4e60}} to {{Gerrit|2d9360cfac}}
* 02:37 ejegg: disabled fundraising CiviCRM jobs for 5.6 update
* 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 13m 58s)


== 2018-09-17 ==
== 2018-10-01 ==
* 23:32 mutante: ms-be2042 - reparing xfs - ([[phab:T199198|T199198]])
* 23:44 eileen: update process control  revision is {{Gerrit|b9c7ab286e}} - define but not enable Redis
* 21:44 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Fighting with scap more (duration: 01m 50s)
* 23:43 foks: disabling 2FA for two users
* 21:42 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Fighting with scap more
* 23:31 twentyafterfour: finished creating database tables
* 21:40 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Fighting with scap (duration: 00m 10s)
* 23:18 twentyafterfour: creating ipblocks_restrictions table (command run on mwmaint2001
* 21:40 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Fighting with scap
* 21:32 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Cleaning up from some by-hand hacks (duration: 00m 10s)
* 21:32 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Cleaning up from some by-hand hacks
* 20:59 andrew@deploy1001: Finished deploy [horizon/deploy@3124052]: Disable unneeded network panels (duration: 03m 34s)
* 20:56 andrew@deploy1001: Started deploy [horizon/deploy@3124052]: Disable unneeded network panels
* 20:49 foks: add email address to User:Lanhiaze
* 20:44 ladsgroup@deploy1001: Finished deploy [ores/deploy@ae96071]: PoolCounter support: Let's get the party started ([[phab:T160692|T160692]]) (duration: 28m 19s)
* 20:24 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@e0b7158]: Update mobileapps to {{Gerrit|d56e4cf}} (duration: 04m 16s)
* 20:19 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@e0b7158]: Update mobileapps to {{Gerrit|d56e4cf}}
* 20:15 ladsgroup@deploy1001: Started deploy [ores/deploy@ae96071]: PoolCounter support: Let's get the party started ([[phab:T160692|T160692]])
* 20:10 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 03m 26s)
* 18:56 stephanebisson: Starting mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki
* 18:55 stephanebisson: Stopped mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki
* 18:46 stephanebisson: Starting mwscript extensions/PageTriage/maintenance/populateDraftQueue.php --wiki enwiki
* 18:40 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T203184|T203184]] (duration: 00m 50s)
* 18:35 sbisson@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/Score/includes/Score.php: [[phab:T203560|T203560]] (duration: 00m 50s)
* 18:14 sbisson@deploy1001: Synchronized static/images/project-logos: SWAT (duration: 00m 50s)
* 17:22 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@6091de6]: Regular WDQS deployment (duration: 11m 45s)
* 17:10 smalyshev@deploy1001: Started deploy [wdqs/wdqs@6091de6]: Regular WDQS deployment
* 16:36 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting wgCommentTableSchemaMigrationStage = WRITE_NEW on test wikis, mw.org ([[phab:T166733|T166733]]) (duration: 00m 50s)
* 16:18 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting wgActorTableSchemaMigrationStage = WRITE_BOTH on test wikis, mw.org ([[phab:T188327|T188327]]) (duration: 00m 50s)
* 15:22 cmjohnson1: db1069 replacing disk in slot 7
* 15:20 jynus: stop db1061 (may generate s6 lag) for hw maintenance
* 15:19 jynus: db1061 (may generate s6 lag) for hw maintenance
* 14:31 banyek: upgrading (kernel & mariadb) db1119
* 14:27 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T203565|T203565]]: depooling db1119 (duration: 00m 49s)
* 13:40 godog: repair sdl on ms-be2041 - [[phab:T199198|T199198]]
* 13:39 moritzm: removed intel-microcode 3.{{Gerrit|20180703}} from apt.wikimedia.org/jessie-wikimedia (superseded by new release shipped via security.debian.org)
* 13:30 moritzm: removed intel-microcode 3.{{Gerrit|20180703}} from apt.wikimedia.org/stretch-wikimedia (superseded by new release shipped via security.debian.org)
* 12:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T203565|T203565]]: depooling db1114 (duration: 00m 49s)
* 12:53 banyek: depooling db1114
* 12:08 jynus: stop and reimage db1071 (may generate s8 lag)
* 12:02 zeljkof: EU SWAT finished
* 12:01 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.20/extensions/CongressLookup/: SWAT: [[gerrit:460860{{!}}Senator John McCain deceased (T203611)]] [[gerrit:460861{{!}}Add Sen. Jon Kyl (T203611)]] (duration: 00m 51s)
* 11:47 moritzm: installing ghostscript security updates on stretch
* 11:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:460864{{!}}Enable WikidataPageBanner for glwiki (T199713)]] (duration: 00m 50s)
* 11:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:460622{{!}}Enable FileExport on Korean Wikipedia (T204399)]] (duration: 00m 50s)
* 11:18 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T204478|T204478]] Disable thumbnail prerendering on private wikis (duration: 00m 49s)
* 11:17 moritzm: installing curl security updates on trusty (Debian already fixed)
* 10:40 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:460873{{!}}Bumping portals to master (T128546)]] (duration: 00m 50s)
* 10:40 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:460873{{!}}Bumping portals to master (T128546)]] (duration: 00m 50s)
* 10:37 addshore: starting my maint script again
* 10:31 addshore: stopped my maint script
* 10:28 addshore: START addshore@mwmaint2001:~$ mwscript refreshLinks.php --wiki wikidatawiki --namespace 146 --e {{Gerrit|56031711}} {{Gerrit|54387042}} # [[phab:T198301|T198301]] [[phab:T195302|T195302]]
* 09:50 marostegui: Deploy schema change on db1062 - [[phab:T203548|T203548]]
* 09:28 marostegui: Deploy schema change on s7 eqiad master (db1062) - [[phab:T187089|T187089]]
* 09:16 jynus: stop and reimage db1068 (may generate s4 lag)
* 08:43 moritzm: installing postgresql security updates on maps*
* 08:29 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:460432{{!}}New throttle rule for enwiki event (T204243)]] (duration: 00m 49s)
* 08:07 jynus: stop and reimage db1062 (may generate s7 lag)
* 07:46 moritzm: repooled wtp1043 ([[phab:T196886|T196886]])
* 07:31 godog: repair sdj on ms-be2040 - [[phab:T199198|T199198]]
* 07:29 godog: force power on for ms-be2030
* 07:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 and db1099:3318 - [[phab:T201011|T201011]] (duration: 00m 49s)
* 07:24 marostegui: Deploy schema change on s8 - [[phab:T201011|T201011]]
* 07:18 marostegui: Stop replication in sync on db1092 and db1099:3318 - [[phab:T201011|T201011]]
* 07:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 and db1099:3318 - [[phab:T201011|T201011]] (duration: 00m 49s)
* 07:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 and db1098:3317 - [[phab:T201011|T201011]] (duration: 00m 49s)
* 06:53 marostegui: Stop replication in sync on db1094 and db1098:3317 - [[phab:T201011|T201011]]
* 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 and db1098:3317 - [[phab:T201011|T201011]] (duration: 01m 04s)
* 06:33 marostegui: Deploy schema change on s7 (frwiktionary,metawiki) - [[phab:T201011|T201011]]
* 06:31 marostegui: Deploy schema change on s4 - [[phab:T201011|T201011]]
* 06:04 marostegui: Deploy schema change on s3 - [[phab:T201011|T201011]]
* 05:47 marostegui: Deploy schema change on s3:testwikidatawiki directly on the master (db2043) - [[phab:T201011|T201011]]
* 05:36 marostegui: Deploy schema change on s3:testwiki directly on the master (db2043) - [[phab:T201011|T201011]]
* 05:28 marostegui: Deploy schema change on s1 eqiad master (db1067) - [[phab:T67448|T67448]] [[phab:T114117|T114117]] [[phab:T51191|T51191]]
* 02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Sep 17 02:46:18 UTC 2018 (duration 10m 52s)
* 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 14m 20s)
 
== 2018-09-15 ==
* 11:57 joal@deploy1001: Finished deploy [analytics/refinery@f4d1f24]: Deploying cluster to prevent failures from missing misc data (duration: 15m 19s)
* 11:42 joal@deploy1001: Started deploy [analytics/refinery@f4d1f24]: Deploying cluster to prevent failures from missing misc data
* 04:53 SMalyshev: re-pooled wdqs2003
* 01:08 SMalyshev: depooled wdqs2003 to let it catch up
 
== 2018-09-14 ==
* 21:19 andrew@deploy1001: Finished deploy [horizon/deploy@56340cd]: Fix proxy creation in neutron regions (duration: 03m 31s)
* 21:16 andrew@deploy1001: Started deploy [horizon/deploy@56340cd]: Fix proxy creation in neutron regions
* 21:03 mutante: ACKed memory error alert on wtp2011 - existing ticket but fresh alert popped up 9h ago ([[phab:T200678|T200678]])
* 20:12 ejegg: updated payments-wiki from {{Gerrit|44b9409104}} to {{Gerrit|43923fadd9}}
* 15:57 jynus: set thread_pool_size to 64 at db2047
* 14:42 ema: lvs3002: restart pybal to remove misc-web [[phab:T164609|T164609]]
* 14:28 ema: lvs2002: restart pybal to remove misc-web [[phab:T164609|T164609]]
* 14:23 ema: lvs1002: restart pybal to remove misc-web [[phab:T164609|T164609]]
* 14:11 jynus: shutting down db1062 for dc maintenance [[phab:T204302|T204302]]
* 13:58 jynus: stop and restart db1118 for upgrade
* 12:51 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: [[phab:T204127|T204127]]: Weight Adjust db2068 (duration: 00m 50s)
* 09:47 banyek: operation stopping db1070 in preparation for reimage
* 08:45 marostegui: Deploy schema change on s7 eqiad master (db1062) - [[phab:T89737|T89737]]
* 08:34 mobrovac@deploy1001: Started restart [cpjobqueue/deploy@32a81be]: (no justification provided)
* 08:29 moritzm: rebooting acamar for kernel tests
* 08:27 elukey: reboot kafka100[1-3] (eventbus eqiad) for kernel upgrades
* 08:20 jynus: stopping and restarting db1069 for upgrade (x1 eqiad master)
* 08:15 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: [[phab:T204127|T204127]]: Weight Adjust db2068 (duration: 00m 50s)
* 07:49 marostegui: Deploy schema change on s4 eqiad master (db1068) - [[phab:T67448|T67448]] [[phab:T114117|T114117]] [[phab:T51191|T51191]]
* 07:40 moritzm: rebooting mwmaint1001 for kernel security update
* 07:28 jynus: stopping db1062 mariadb in preparation for reimage
* 07:11 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: [[phab:T204127|T204127]]: Weight Adjust db2068 (duration: 00m 50s)
* 05:34 marostegui: Deploy schema change on s3 eqiad master - [[phab:T187089|T187089]]
* 05:29 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2050 - [[phab:T189101|T189101]] (duration: 00m 49s)
* 05:17 marostegui: Stop replication in sync on db1075 and db2050 - [[phab:T189101|T189101]]
* 05:16 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2050 - [[phab:T189101|T189101]] (duration: 00m 49s)
* 05:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - [[phab:T204127|T204127]] (duration: 00m 50s)
* 05:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2068 - [[phab:T204127|T204127]] (duration: 00m 52s)
 
== 2018-09-13 ==
* 21:55 urandom: running fstrim --all on restbase1013 -- [[phab:T89584|T89584]]
* 21:51 urandom: running fstrim --all on restbase1008 -- [[phab:T89584|T89584]]
* 21:35 urandom: running fstrim --all on restbase1011 -- [[phab:T89584|T89584]]
* 21:33 urandom: running fstrim --all on restbase1007 -- [[phab:T89584|T89584]]
* 19:31 herron: shutting down old puppet compiler instances in puppet3-diffs project [[phab:T191438|T191438]]
* 19:24 herron: moving puppet-compiler.wmlabs.org proxy from puppet3-diffs to puppet-diffs project [[phab:T191438|T191438]]
* 18:40 cmjohnson1: updating f/w cloudvirt1019 disabled icinga checks
* 17:42 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/resources/src/mediawiki.widgets/mw.widgets.CheckMatrixWidget.js: {{Gerrit|I1f92479bf1}}, [[phab:T203325|T203325]] (duration: 00m 50s)
* 17:41 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/widget/CheckMatrixWidget.php: {{Gerrit|I1f92479bf1}}, [[phab:T203325|T203325]] (duration: 00m 49s)
* 17:40 krinkle@deploy1001: Synchronized php-1.32.0-wmf.20/includes/htmlform/fields/HTMLCheckMatrix.php: {{Gerrit|I1f92479bf1}}, [[phab:T203325|T203325]] (duration: 00m 51s)
* 17:38 XioNoX: enable cr2:xe-4/0/0 (to asw-a) - [[phab:T203719|T203719]]
* 17:31 XioNoX: disable cr2:xe-4/0/0 (to asw-a) for optics replacement (round 2, 1st one didn't clear the errors, need to do the other side) - [[phab:T203719|T203719]]
* 17:29 mutante: radium - removing tor package, clearing systemd failed units, to clear Icinga alerts from this host that is to be decomed
* 17:29 XioNoX: enable cr2:xe-4/0/0 (to asw-a) for optics replacement - [[phab:T203719|T203719]]
* 17:25 XioNoX: disable cr2:xe-4/0/0 (to asw-a) for optics replacement - [[phab:T203719|T203719]]
* 16:45 godog: repair sde on ms-be1041 - [[phab:T199198|T199198]]
* 15:26 jynus: restarting again db1075 for proper kernel upgrade
* 15:25 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - [[phab:T204127|T204127]] (duration: 00m 49s)
* 15:15 herron: repool mx1001 — upgrade to stretch complete [[phab:T175361|T175361]]
* 15:02 akosiaris: upload blubber_0.5.0-1 to apt.wikimedia.org/{stretch,jessie}-wikimedia/main [[phab:T203121|T203121]]
* 14:47 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - [[phab:T204127|T204127]] (duration: 00m 50s)
* 14:36 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - https://phabricator.wikimedia.org/T174047
* 14:35 bstorm_: switch labstore1004 and labstore1005 to using the cfq scheduler on the DRBD volumes
* 14:32 moritzm: rebooting dns1002 for kernel security update
* 13:59 herron: depool mx1001 and relay queued messages to mx2001 for upgrade to stretch [[phab:T175361|T175361]]
* 13:51 moritzm: rebooting dns1001 for kernel security update
* 13:28 andrew@deploy1001: Finished deploy [horizon/deploy@c9c7a56]: Improvements for VM creation in eqiad1, [[phab:T167293|T167293]] (take two) (duration: 04m 10s)
* 13:24 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Increase weight for db2054 - [[phab:T204127|T204127]] (duration: 00m 49s)
* 13:23 andrew@deploy1001: Started deploy [horizon/deploy@c9c7a56]: Improvements for VM creation in eqiad1, [[phab:T167293|T167293]] (take two)
* 13:20 andrew@deploy1001: Finished deploy [horizon/deploy@12aa2d3]: Improvements for VM creation in eqiad1, [[phab:T167293|T167293]] (duration: 00m 13s)
* 13:20 andrew@deploy1001: Started deploy [horizon/deploy@12aa2d3]: Improvements for VM creation in eqiad1, [[phab:T167293|T167293]]
* 13:18 marostegui: Deploy schema change on s2 eqiad master (db1066) - [[phab:T89737|T89737]]
* 13:07 marostegui: Deploy schema change on s8 eqiad master (db1071) - [[phab:T187089|T187089]]
* 13:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2085:3318 - [[phab:T189101|T189101]] (duration: 00m 49s)
* 12:55 marostegui: Stop db1071 (s8 eqiad master) and db2085:3318 in sync - [[phab:T189101|T189101]]
* 12:54 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085:3318 - [[phab:T189101|T189101]] (duration: 00m 49s)
* 12:45 marostegui: Reload haproxy on dbproxy1011 to repool labsdb1009 - https://phabricator.wikimedia.org/T174047
* 11:47 moritzm: rebooting dns recursors in codfw for kernel security update
* 11:38 jynus: stopping eqiad s3 replication and shuttinf down db1075 in preparation for reimage [[phab:T148507|T148507]]
* 10:49 banyek: shutting down db2068 and dbstore2001:s7 for cloning
* 10:46 moritzm: uploaded debdeploy 0.0.99.5-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
* 10:33 marostegui: Deploy schema change on s4 eqiad master (db1068) - [[phab:T187089|T187089]]
* 10:33 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Slowly repool db2054 - [[phab:T204127|T204127]] (duration: 00m 50s)
* 09:57 marostegui: Deploy schema change on s4 eqiad master (db1068) - [[phab:T89737|T89737]]
* 09:54 godog: repair sdd on ms-be2043 - [[phab:T199198|T199198]]
* 09:44 marostegui: Enable GTID on eqiad masters - [[phab:T189107|T189107]]
* 09:41 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp2020.codfw.wmnet
* 09:34 moritzm: rebooting dns recursors in esams for kernel security update
* 09:24 marostegui: Reload haproxy on dbproxy1011 to depool labsdb1009 - https://phabricator.wikimedia.org/T174047
* 09:08 marostegui: Deploy schema change on s8 eqiad master (db1071) - [[phab:T89737|T89737]]
* 08:59 hashar: Bumped CI jobs based on Debian Stretch to use Chrome 69 and Firefox 60 {{!}} [[phab:T203902|T203902]]
* 08:51 moritzm: rebooting dns recursors in ulsfo for kernel security update
* 08:49 marostegui: Deploy schema change on s5 eqiad master (db1070) - [[phab:T89737|T89737]]
* 08:41 moritzm: removed labvirt1019/labvirt1020 from debmonitor ([[phab:T204004|T204004]])
* 08:36 marostegui: Deploy schema change on s6 eqiad master (db1061) - [[phab:T89737|T89737]]
* 08:29 moritzm: installing curl security updates
* 08:21 marostegui: Deploy schema change on s6 eqiad master (db1061) - [[phab:T187089|T187089]]
* 08:16 marostegui: Stop replication on s4 eqiad master (db1068) and deploy a schema change - this will generate lag on s4 eqiad - [[phab:T144010|T144010]]
* 08:08 marostegui: Deploy schema change on s5 eqiad master - [[phab:T187089|T187089]]
* 08:01 moritzm: rebooting dns recursors in eqsin for kernel security update
* 08:01 marostegui: Disconnect replication eqiad -> codfw on s1-s8, x1, es2, es3 - [[phab:T189107|T189107]]
* 07:47 elukey: reboot kafka10[12-23] (old analytics cluster) for kernel upgrades
* 07:46 elukey: execute apt-get autoremove on notebook* to remove old nginx packages (not used anymore)
* 07:25 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Tune db weights even more (duration: 00m 50s)
* 06:56 elukey: reboot notebook100[3,4] for kernel upgrades - [[phab:T203165|T203165]]
* 06:42 elukey: reboot stat100[4-6] for kernel upgrades - [[phab:T203165|T203165]]
* 05:11 marostegui: Stop MySQL on db2054 and dbstore2001:3317 to clone db2054 - [[phab:T204127|T204127]]
* 02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Sep 13 02:46:16 UTC 2018 (duration 10m 46s)
* 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.20) (duration: 14m 03s)
* 02:24 ejegg: restarted fundraising CiviCRM jobs
* 01:48 ejegg: updated CiviCRM from {{Gerrit|2a3583cca1}} to {{Gerrit|b0d7df4e60}}
* 01:45 ejegg: disabled fundraising CiviCRM jobs
* 00:00 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7e5e537]: Deploy Blazegraph & Updater for [[phab:T202765|T202765]] and [[phab:T203646|T203646]] handling (duration: 23m 45s)
 
== 2018-09-12 ==
* 23:36 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7e5e537]: Deploy Blazegraph & Updater for [[phab:T202765|T202765]] and [[phab:T203646|T203646]] handling
* 22:41 Trey314159: reindexing Esperanto wikis on elastic@codfw and elastic@eqiad complete ([[phab:T203005|T203005]])
* 19:50 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease db2061 load (duration: 00m 50s)
* 19:13 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Decrease db2065 load (duration: 00m 49s)
* 18:24 Trey314159: reindexing Esperanto wikis on elastic@codfw and elastic@eqiad ([[phab:T203005|T203005]])
* 17:59 otto@deploy1001: Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars  for [[phab:T203804|T203804]] (duration: 06m 11s)
* 17:53 otto@deploy1001: Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars  for [[phab:T203804|T203804]]
* 17:52 otto@deploy1001: Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars  for [[phab:T203804|T203804]] (duration: 01m 41s)


__NOTOC__
__NOTOC__
{{SAL-archives}}
==Archives==
See [[Server admin log/Archives]].
<noinclude>[[Category:SAL]][[Category:Operations]]</noinclude>
<noinclude>[[Category:SAL]][[Category:Operations]]</noinclude>

Revision as of 16:53, 2 October 2018

2018-10-02

  • 16:53 jynus: setup test s3 replication channel on db1110 (filtered)
  • 16:49 XioNoX: assign 10.195.0.129/29 to pfw3-codfw:reth0.2133 - T204271
  • 16:38 cmjohnson1: swapping failed disk db1067 T205780
  • 16:04 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@093551f]: Increase cirrusSearchLinksUpdate concurrency (duration: 01m 06s)
  • 16:03 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@093551f]: Increase cirrusSearchLinksUpdate concurrency
  • 15:50 marxarelli: cutting 1.32.0-wmf.24 branch
  • 15:33 gehel: cleanup old cronjob (cleanup GC logs) on all elasticsearch servers
  • 15:24 akosiaris: upgrade mathoid chart version to 0.0.11
  • 15:24 akosiaris@deploy1001: scap-helm mathoid finished
  • 15:23 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 15:23 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 15:23 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
  • 15:21 akosiaris@deploy1001: scap-helm mathoid finished
  • 15:21 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 15:21 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 15:21 akosiaris@deploy1001: scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw]
  • 14:11 banyek: powering off dbstore2002.codfw.wmnet for BBU change (T205257)
  • 13:47 marostegui: Deploy schema change on s4 eqiad, this will generate lag on eqiad - T205913
  • 13:06 marostegui: Deploy schema change on s7 eqiad, this will generate lag on eqiad - T205913
  • 12:47 banyek: converting enwiki.content to TokuDB on host dbstrore1002 (T205544)
  • 12:47 banyek: converting enwiki.contents to TokuDB on host dbstrore1002 (T205544)
  • 11:58 banyek: converting wikidatawiki.slots to TokuDB on host dbstrore1002 (T205544)
  • 11:41 arturo: downtime labstore1007 load check in icinga for 1d
  • 11:21 zeljkof: EU SWAT finished
  • 11:19 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: SWAT: Use proper index on change_tag table (T205904) (duration: 00m 57s)
  • 10:58 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: Delay job execution while in read-only mode - T204154 (duration: 00m 57s)
  • 10:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2092 (duration: 00m 56s)
  • 10:24 marostegui: Deploy schema change on db2092 - T203709
  • 10:24 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2092 (duration: 00m 56s)
  • 09:30 marostegui: Deploy schema change on s2 eqiad master, lag will be generated T205913
  • 08:43 banyek: disabling puppet on es2001 and disabling backups too
  • 08:28 marostegui: Deploy schema change on s6 eqiad master, lag will be generated T205913
  • 08:16 jynus: test recover some s3 wiki data onto db1110 (s5)
  • 08:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 56s)
  • 08:04 marostegui: Deploy schema change on s5 eqiad master, lag will be generated T205913
  • 08:01 banyek: converting wikidatawiki.content to TokuDB on host dbstrore1002 (T205544)
  • 07:54 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 55s)
  • 07:50 marostegui: Deploy schema change on db2071 T205913
  • 07:50 mholloway-shell@deploy1001: Finished deploy [tilerator/deploy@6c80537] (maps1004): Disable event logging requests and remove HTTP proxy (duration: 00m 17s)
  • 07:49 mholloway-shell@deploy1001: Started deploy [tilerator/deploy@6c80537] (maps1004): Disable event logging requests and remove HTTP proxy
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 56s)
  • 07:48 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@0bf513a] (maps1004): Remove HTTP proxy (duration: 00m 16s)
  • 07:48 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@0bf513a] (maps1004): Remove HTTP proxy
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 56s)
  • 07:36 marostegui: Deploy schema change on db2088:3311 T205913
  • 07:36 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 55s)
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
  • 07:18 marostegui: Deploy schema change on db2072 T205913
  • 07:17 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 01m 02s)
  • 05:22 _joe_: stopped tilerator on maps1004, was spamming like crazy
  • 01:18 ejegg: updated CiviCRM from e7a620a00c to c353eba283

2018-10-01

  • 23:44 eileen: update process control revision is b9c7ab286e - define but not enable Redis
  • 23:43 foks: disabling 2FA for two users
  • 23:31 twentyafterfour: finished creating database tables
  • 23:18 twentyafterfour: creating ipblocks_restrictions table (command run on mwmaint2001: foreachwiki sql.php maintenance/archives/patch-ipblocks_restrictions-table.sql)
  • 22:52 ppchelko@deploy1001: Finished deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 3, feeds check timeouts (duration: 06m 22s)
  • 22:46 ppchelko@deploy1001: Started deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 3, feeds check timeouts
  • 22:45 ppchelko@deploy1001: Finished deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 2, feeds check timeouts (duration: 03m 57s)
  • 22:41 ppchelko@deploy1001: Started deploy [restbase/deploy@babfe80]: Don't log the request for transform failures, take 2, feeds check timeouts
  • 22:41 ppchelko@deploy1001: Finished deploy [restbase/deploy@babfe80]: Don't log the request for transform failures (duration: 12m 27s)
  • 22:29 ppchelko@deploy1001: Started deploy [restbase/deploy@babfe80]: Don't log the request for transform failures
  • 21:17 arlolra: Updated Parsoid to 224ecde (T198504, T133673, T202666)
  • 20:45 arlolra@deploy1001: Finished deploy [parsoid/deploy@8ff45db]: Updating Parsoid to 224ecde (duration: 08m 22s)
  • 20:37 arlolra@deploy1001: Started deploy [parsoid/deploy@8ff45db]: Updating Parsoid to 224ecde
  • 20:35 gehel@deploy1001: Finished deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph (duration: 14m 00s)
  • 20:21 gehel@deploy1001: Started deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph
  • 19:52 gehel@deploy1001: Finished deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph (wdqs1009 only) (duration: 00m 30s)
  • 19:51 gehel@deploy1001: Started deploy [wdqs/wdqs@a637583]: New version of WDQS GUI, updater and blazegraph (wdqs1009 only)
  • 19:27 ppchelko@deploy1001: Finished deploy [restbase/deploy@7caf4d8]: Content-negotiation filter going live T128040 (duration: 03m 38s)
  • 19:24 ppchelko@deploy1001: Started deploy [restbase/deploy@7caf4d8]: Content-negotiation filter going live T128040
  • 19:11 thcipriani: restarting ci jenkins for new plugins
  • 18:33 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable page issues A/B test at 20% rate (T200792) (duration: 00m 56s)
  • 18:28 Amir1: ladsgroup@mwmaint2001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=enwiki --prefix (T201009)
  • 18:23 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/maintenance/includes/DeleteLocalPasswords.php: T201009 (duration: 00m 56s)
  • 18:17 catrope@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/PageTriage/: Ensure valid AFC option is selected (T205324, T205168); hide copyvio behind a global var and URL param (duration: 00m 57s)
  • 18:12 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable page issues A/B test at 5% rate (T200792) (duration: 00m 59s)
  • 17:59 XioNoX: push fw change on pfw3-eqiad - T205888
  • 17:57 XioNoX: push fw change on pfw3-codfw - T205888
  • 17:28 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@a637583]: Test deployment for recent updater build and GUI changes. Also blazegraph updates(wdqs1009) (duration: 01m 46s)
  • 17:27 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@a637583]: Test deployment for recent updater build and GUI changes. Also blazegraph updates(wdqs1009)
  • 17:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093, db1064 (duration: 00m 57s)
  • 17:02 jynus: stopping some mariadb instances on dbstore1001 and starting compression T201392
  • 16:26 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@58f9ed3]: Fix KafkaConsumer not connected error
  • 15:16 jynus: stopping db1064 to clone it to dbstore1001
  • 15:00 akosiaris: upgrade etherpad to 1.7.0-2
  • 14:14 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR migration stage to write-both/read-new on mediawikiwiki (T198308) (duration: 00m 56s)
  • 13:51 banyek: Downtimed the slave lag monitoring on dbstore1002 while the tables getting converted (T205544)
  • 12:38 akosiaris: upload hfst_3.13.0~r3461-1+wmf2 to apt.wikimedia.org/jessie-wikimedia/main. T199962
  • 12:26 banyek: converting enwiki.categorylinks to TokuDB on host dbstrore1002 (T205544)
  • 12:19 banyek: stopping replication on s2@dbstore20002: the tables being compressed (T204930)
  • 12:19 banyek: stopping replication on s2@dbstore20002: the tables being compressed
  • 12:15 banyek: enabling puppet on labsdb1009, labsdb1010, labsdb1011 (T183983)
  • 12:13 zeljkof: EU SWAT finished
  • 12:12 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.23/extensions/ContentTranslation/: SWAT: Fix error in CXTransclusionNode#afterRender method (T205521) (duration: 00m 59s)
  • 11:56 jynus: stopping db1093 to clone it to dbstore1001
  • 11:52 arturo: install prometheus-openstack-exporte 0.0.8-3 in reprepro T203177
  • 11:41 zfilipin@deploy1001: Synchronized wmf-config: SWAT: Remove unused default source language config for CX (duration: 00m 57s)
  • 11:16 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2058 (duration: 00m 55s)
  • 11:09 _joe_: killed bash runner.sh by user ladsgroup on mwmaint2001
  • 10:58 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2058 (duration: 00m 57s)
  • 10:52 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093, db1064 (duration: 00m 57s)
  • 10:42 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 56s)
  • 10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 59s)
  • 10:21 godog: repair /dev/sdf1 /dev/sde1 on ms-be1041 - T199198
  • 10:15 Amir1: ladsgroup@mwmaint2001:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --prefix on all CentralAuth wikis (T201009)
  • 10:10 Amir1: mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki --delete (T201009)
  • 09:33 godog: test formatting sdh and sdi on ms-be2040 with crc=0 - T199198
  • 09:15 volans: Set Racktables in read-only mode - T199083
  • 08:56 _joe_: rolling restart of parsoid in codfw; afterwards, parsoid will connect to the MediaWiki API via HTTPS
  • 08:54 _joe_: rolling restart of parsoid in eqiad
  • 07:54 banyek: disabling puppet on labsdb1009, labsdb1010, labsdb1011 (T183983)
  • 07:54 banyek: disabling puppet on labsdb1009, labsdb1010, labsdb1011
  • 07:00 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462) (duration: 00m 16s)
  • 07:00 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462)
  • 06:39 mholloway-shell@deploy1001: Finished deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462) (duration: 00m 19s)
  • 06:39 mholloway-shell@deploy1001: Started deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462)
  • 05:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 (duration: 00m 56s)
  • 05:19 marostegui: Stop replication on dbstore1002 and db1103:3312 in sync
  • 05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 (duration: 01m 01s)
  • 05:19 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@07cbfb4]: Update mobileapps to a1fa41b (duration: 03m 18s)
  • 05:15 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@07cbfb4]: Update mobileapps to a1fa41b
  • 05:07 marostegui: Deploy schema change on s1 codfw msater - T203709
  • 03:21 onimisionipe: restarting inplace reindexing of enwiki and viwiki at codfw - T204362


Archives

See Server admin log/Archives.