You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log

From Wikitech-static
Revision as of 21:46, 12 January 2019 by imported>Stashbot (akosiaris: restart all zotero pods in eqiad)
Jump to navigation Jump to search

2019-01-12

  • 21:46 akosiaris: restart all zotero pods in eqiad
  • 16:12 moritzm: rebooting mw2167 for a test
  • 02:16 legoktm@deploy1001: Synchronized docroot/mediawiki.org/keys: Add Mukunda's new subkey that was used for the 1.32 release - T213521 (duration: 00m 47s)

2019-01-11

  • 21:56 jforrester@deploy1001: Finished scap: Full scap sync to update wmf.12 i18n for the weekend Idf2a67860f (duration: 19m 12s)
  • 21:37 jforrester@deploy1001: Started scap: Full scap sync to update wmf.12 i18n for the weekend Idf2a67860f
  • 18:43 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: Update ExtensionDistributor for 1.32 release - https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/483735 (duration: 00m 46s)
  • 18:07 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2060 T210713 (duration: 00m 46s)
  • 17:10 marostegui: Deploy schema change on db2060 - T210713
  • 16:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2060 T210713 (duration: 00m 46s)
  • 16:53 marostegui: Defragment change_tag table on db2060 - T210713
  • 14:37 jynus: upgrade and restart db2091 (s2, s4)
  • 14:12 jynus: updating mariadb client packages on cumin* hosts
  • 11:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: repool es1018 fully (duration: 00m 46s)
  • 11:21 jynus: stop, upgrade and reboot es2017
  • 11:04 jynus: stop, upgrade and reboot es2016
  • 10:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: repool es1018 with low load (duration: 00m 46s)
  • 10:31 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2013 (duration: 00m 45s)
  • 10:30 jynus: upgrade and restart es1018
  • 09:58 jynus: upgrade and reboot es2013
  • 09:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2013 (duration: 00m 45s)
  • 09:49 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2013 (duration: 00m 47s)
  • 09:32 jynus: reset iLo on db2053
  • 08:49 moritzm: installing tmpreaper security updates
  • 02:40 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ib87407165382 (duration: 00m 46s)
  • 01:20 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T211993 Enable GrowthExperiments help panel for 50% of new users on cswiki and kowiki (duration: 00m 46s)
  • 01:05 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T211993 Enable GrowthExperiments help panel on cswiki and kowiki (duration: 00m 45s)
  • 01:03 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/WikimediaEvents/includes/PageViews.php: SWAT: T213186 GrowthExperiments: Support templates for help desk title (duration: 00m 46s)
  • 00:50 XioNoX: bump prefix limit for AS6939 in eqsin
  • 00:18 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/AbuseFilter/includes/AbuseFilterHooks.php: T213453: Use slot in onEditFilterMergedContent and newVariableHolderForEdit in AbuseFilter (duration: 00m 47s)
  • 00:12 James_F: 482373 is live on mwdebug1002 for extensive checks.
  • 00:08 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Help panel: Set help desk page correctly on kowiki Ia94cfc571 (duration: 00m 46s)

2019-01-10

  • 23:45 Krinkle: krinkle@tungsten: upgrade xhgui to include upstream f039fb9f99f - T213218
  • 23:45 Krinkle: upgraded xhgui to upstream 2965240c91e52 (current upstream master) - T213218
  • 23:36 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T213497 [Commons, TestCommons] Don't use Wikibase entity search (duration: 00m 46s)
  • 22:57 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/includes/EditEntity/MediawikiEditFilterHookRunner.php: T213453: Pass slotrole into EditFilterMergedContent hook in Wikibase repo (duration: 00m 47s)
  • 20:47 marxarelli: both mediawiki error rates and 500 response rates have subsided back to pre-deploy levels
  • 20:19 marxarelli: seeing increase in "60 second timed out" error rate and rise in 503 rate, as was the case with group1 deployment. continuing to monitor
  • 20:11 gehel: restart blazegraph on wdqs1009 to validate new config
  • 20:02 tgr@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/WikimediaEvents/modules/ve-wme/campaigns.js: SWAT: Remove unnecessary addPlugin wrapper (T213338) (duration: 00m 53s)
  • 19:50 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove AICaptcha settings (T186244) (duration: 00m 52s)
  • 19:47 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Whitelist *.*.archive.org in wgCopyUploadsDomains (T207581) (duration: 00m 53s)
  • 19:41 tgr: ran mwscript namespaceDupes.php bnwikibooks --fix (238 links fixed)
  • 19:41 volans: installed spicerack 0.0.12-1 on cumin2001 T205884
  • 19:39 volans: uploaded spicerack_0.0.12-1_amd64.deb to apt.wikimedia.org stretch-wikimedia T205884
  • 19:39 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Note that namespaceDupes.php maintenance script run will be needed after the deployment. (T203534) (duration: 00m 53s)
  • 19:14 marostegui: Deploy schema change on dbstore1001 - T85757
  • 19:13 marostegui: Deploy schema change on dbstore1002 - T85757
  • 18:57 tzatziki: deleting three files for legal compliance
  • 18:52 anomie@mwmaint1002: Running migrateActors.php on test wikis and mediawikiwiki for T188327. This may cause lag in codfw.
  • 18:47 marostegui: Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T85757
  • 18:46 marostegui: Stop replication on s1 codfw master for a schema change - T85757
  • 18:37 marostegui: Stop replication on s8 codfw master for a schema change - T85757
  • 18:30 marostegui: Upgrade mysql and kernel on db2060
  • 18:30 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2053, db2060 for kernel and mysql upgrade (duration: 00m 51s)
  • 18:13 marostegui: Stop MySQL on db2046 for kernel upgrade
  • 18:12 marostegui: The above change was db2053 and not db2060
  • 18:11 marostegui: Stop MySQL on db2053 and db2060 for mysql and kernel upgrade
  • 18:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2053, db2060 for kernel and mysql upgrade (duration: 00m 53s)
  • 17:50 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2015 (duration: 00m 53s)
  • 17:49 marostegui: Deploy schema change on db2053 - T210713
  • 17:33 marostegui: Deploy schema change on db2046 - T210713
  • 16:59 jynus: stop and upgrade es2015
  • 16:52 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2015 (duration: 00m 52s)
  • 16:41 onimisionipe: data transfer from wdqs1004 -> wdqs1006 completed! - T213361
  • 16:32 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T159708 Enable Structured Data on Commons, captions-only (duration: 00m 53s)
  • 16:17 James_F: T180981 Placed patch to enable WBMI on Commons on mwdebug1002
  • 16:13 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T180981 Add Commons to wikis with WikibaseMediaInfo installed (duration: 00m 52s)
  • 16:11 jforrester@deploy1001: Synchronized dblists/wikidatarepo.dblist: T180981 Add Commons to wikis with WikibaseRepo installed (duration: 00m 54s)
  • 16:04 James_F: T180981 Placed patch to install but not enable WBMI on Commons on mwdebug1002
  • 15:56 marostegui: Deploy schema change on db1068 (s4 master) - T86338
  • 15:31 fsero: rollbacking last zotero codfw deployment
  • 15:27 marostegui: Deploy schema change on db1067 (s1 master) - T86338 T202167
  • 15:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 T86338 T202167 (duration: 00m 49s)
  • 15:24 addshore: T208330, MariaDB [testcommonswiki]> TRUNCATE TABLE wb_terms; # Was https://phabricator.wikimedia.org/P7973
  • 15:22 fsero@deploy1001: scap-helm zotero upgrade production -f /srv/scap-helm/zotero/zotero-values-codfw.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: codfw]
  • 15:21 fsero@deploy1001: scap-helm zotero upgrade -f /srv/scap-helm/zotero/zotero-values-codfw.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: codfw]
  • 15:20 addshore@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/includes/Content: T208330 dont write to wb_terms for mediainfo (duration: 00m 54s)
  • 15:12 addshore@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Wikibase/repo/includes/Content: T208330 dont write to wb_terms for mediainfo (duration: 00m 55s)
  • 14:59 marostegui: Deploy schema change on db1080 - T86338 T202167
  • 14:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 T86338 T202167 (duration: 00m 52s)
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 T86338 T202167 (duration: 00m 52s)
  • 14:42 fsero@deploy1001: scap-helm zotero finished
  • 14:42 fsero@deploy1001: scap-helm zotero cluster staging completed
  • 14:42 fsero@deploy1001: scap-helm zotero upgrade staging -f /srv/scap-helm/zotero/zotero-values-staging.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: staging]
  • 14:36 fsero@deploy1001: scap-helm zotero finished
  • 14:36 fsero@deploy1001: scap-helm zotero cluster staging completed
  • 14:36 fsero@deploy1001: scap-helm zotero upgrade staging -f /srv/scap-helm/zotero/zotero-values-staging.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: staging]
  • 14:35 fsero@deploy1001: scap-helm zotero upgrade staging -f /srv/scap-helm/zotero/zotero-values-staging.yaml [namespace: zotero, clusters: staging]
  • 14:33 fsero@deploy1001: scap-helm -h finished
  • 14:33 fsero@deploy1001: scap-helm -h cluster staging completed
  • 14:33 fsero@deploy1001: scap-helm -h [namespace: -h, clusters: staging]
  • 14:33 marostegui: Deploy schema change on db1114 - T86338 T202167
  • 14:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 T86338 T202167 (duration: 00m 53s)
  • 14:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: depool es1019 (duration: 00m 53s)
  • 13:51 arturo: T212302 icinga downtime for 2h cloudvirt[1013,1024,1026-1030].eqiad.wmnet bc wrong puppet code
  • 13:24 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: depool es1018 (duration: 00m 52s)
  • 13:10 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2012 (duration: 00m 52s)
  • 13:01 zeljkof: EU SWAT finished
  • 13:01 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: Remove main page special casing from ruwikibooks and ruwikiquote (T212849) (duration: 00m 52s)
  • 12:58 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: Remove main page special casing from eswiki (T212849) (duration: 00m 53s)
  • 12:53 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: Turn off main page special casing for svwiki (T213018) (duration: 00m 52s)
  • 12:46 zfilipin@deploy1001: Synchronized dblists/flow.dblist: SWAT: Disable unused Flow extension on ur.wikibooks (T207627) (duration: 00m 55s)
  • 12:42 onimisionipe: starting data transfer from wdqs1004 -> wdqs1006 - T213361
  • 12:34 onimisionipe: starting data transfer from wdqs1003 -> wdqs1006 - T213361 - aborted (nodes are in different cluster)
  • 12:28 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Re-enable QuickSurveys extension on enwiki (T209882) (duration: 00m 52s)
  • 12:20 jynus: stop and upgrade es2012
  • 12:12 zfilipin@deploy1001: Synchronized dblists/flow.dblist: SWAT: Reverted "Revert "Disable unused Flow extension on de.wikiversity"" (T207626) (duration: 00m 53s)
  • 12:01 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2012 (duration: 00m 52s)
  • 11:54 onimisionipe: starting data transfer from wdqs1003 -> wdqs1006 - T213361
  • 10:59 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209857 Increase CPU benchmark sampling rate (duration: 00m 53s)
  • 10:58 fsero: uploaded docker-registry_2.7.0~rc0~wmf1-1 debian package to reprepro for stretch-wikimedia (done yesterday at 17:21 UTC forgot about the log)
  • 10:26 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209857 Run CPU benchmark for a portion of navtiming pageloads (duration: 00m 52s)
  • 10:10 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209857 Run CPU benchmark for a portion of navtiming pageloads (duration: 00m 53s)
  • 09:52 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 Decrease ruwiki navtiming rate (duration: 00m 52s)
  • 09:45 gilles@deploy1001: Synchronized tests/InitialiseSettingsTest.php: T211395 T211529 tests: Assert that extra namespaces have correspondent talk namespaces (duration: 00m 56s)
  • 09:34 moritzm: updated thirdparty/php72 component for stretch-wikimedia to 7.2.13
  • 01:41 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make GrowthExperiments config wmf.12-proof (duration: 00m 52s)
  • 01:21 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert latest config patch (caused fatal errors on kowiki) (duration: 00m 52s)
  • 00:58 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure help desk page for help panel correctly on kowiki (T213186) (duration: 00m 53s)
  • 00:56 cstone: updated fundraising tools from 5f44d9dd43 to da82ed111d
  • 00:34 catrope@deploy1001: Synchronized php-1.33.0-wmf.12/includes/MovePage.php: Fix missing ATOMIC_CANCELABLE in MovePage::move() (T213168) (duration: 00m 53s)
  • 00:20 catrope@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/GrowthExperiments/: Help panel fixes (T212973, T212890, T213186) (duration: 00m 54s)
  • 00:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable EventLogging for GrowthExperiments help panel (T211991) (duration: 00m 54s)

2019-01-09

  • 23:51 mutante: thumb1004 - still needs broken RAM replaced, expired downtime, re-ACKed (T207721)
  • 23:39 mutante: mw2151 - change netbox status from active to staged - it's not actually active, it's role(spare) and was jessie (T192457)
  • 23:34 mutante: reinstalling mw2151.codfw.wmnet because it was the very last mw* host on jessie
  • 21:20 bblack: multatuli (ns2) - upgrade gdnsd to 9949 beta release
  • 21:04 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@bfa9241]: Increase concurrency for categoryMembershipJob T192691 (duration: 00m 45s)
  • 21:04 James_F: Creating Wikibase repo tables on Commons for T68108
  • 21:03 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@bfa9241]: Increase concurrency for categoryMembershipJob T192691
  • 21:00 James_F: Running rebuildall on TestCommons
  • 20:53 bblack: authdns1001 (ns0) - upgrade gdnsd to 9949 beta release
  • 20:45 James_F: Created Wikibase repo tables on TestCommons
  • 20:11 dduvall@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.12 (duration: 00m 53s)
  • 20:10 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.12
  • 19:28 crusnov@deploy1001: Finished deploy [netbox/deploy@7fe39e1]: Deploy Django security upgrade (duration: 04m 33s)
  • 19:23 crusnov@deploy1001: Started deploy [netbox/deploy@7fe39e1]: Deploy Django security upgrade
  • 19:01 ejegg: updated standalone SmashPig deploy from 25713ca232 to 78b92b7fef
  • 18:43 bblack: authdns2001 (ns1) - upgrade gdnsd to 9949 beta release
  • 18:26 XioNoX: add bgp sessions to AS31800 on cr1-eqsin
  • 18:19 marostegui: Rename table tag_summary on enwiki on db1089 - T212255
  • 18:18 XioNoX: add bgp sessions to AS38895 on cr1-eqsin
  • 18:04 marostegui: Drop valid_tag from s3 master (db1075) - T212254
  • 17:39 tarrow: That last one was SWAT: T209504 Increase PHP constraint check entities to 150
  • 17:36 tarrow@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 53s)
  • 17:28 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - T86338
  • 17:18 James_F: Ran `namespaceDupes.php --wiki=bewikibooks` on mwmaint1002, no change
  • 17:16 bblack: uploaded gdnsd-2.99.9949-beta-1+wmf1 to reprepro for stretch-wikimedia
  • 17:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 T86338 T202167 (duration: 00m 52s)
  • 16:29 marostegui: Deploy schema change on db1083 - T86338 T202167
  • 16:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 T86338 T202167 (duration: 00m 53s)
  • 16:17 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 with full weight (duration: 00m 53s)
  • 16:11 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: T213227 RepoHooks::onApiCheckCanExecute: Only fail if the edit is for our entity's slot (duration: 00m 54s)
  • 15:50 marostegui: Drop valid_tag tables from db1095 (s3) - T212254
  • 15:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 T86338 T202167 (duration: 00m 51s)
  • 15:23 jijiki: restarting scb* pdfrender
  • 15:10 marostegui: Deploy schema change on db1106 (sanitarium s1 master) with replication, lag will be generated on s1 labs - T86338 T202167
  • 15:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 T86338 T202167 (duration: 00m 52s)
  • 14:39 elukey: restart Hadoop HDFS namenodes on an-master100[1,2] to complete decom of analytics1028->41
  • 14:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 T212254 (duration: 00m 53s)
  • 14:36 volans@deploy1001: Finished deploy [debmonitor/deploy@0f096de]: Deploy Django security upgrade (duration: 01m 50s)
  • 14:34 volans@deploy1001: Started deploy [debmonitor/deploy@0f096de]: Deploy Django security upgrade
  • 14:28 marostegui: valid_tag table on db1077 with replication (lag will be generated on labs s3) - T212254
  • 14:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 T212254 (duration: 00m 52s)
  • 13:32 urandom: forcing removal of restbase1016-c (host down way too long to salvage) -- T212418
  • 13:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight (duration: 00m 52s)
  • 13:26 zeljkof: EU SWAT finished
  • 13:22 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/: SWAT: Fix order of arguments in ChangeTags::getPrevTags ([T212703]) (duration: 05m 50s)
  • 13:08 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.12/: SWAT: Fix order of arguments in ChangeTags::getPrevTags ([T212703]) (duration: 06m 54s)
  • 13:00 zeljkof: extending eu swat for 5-10 minutes
  • 12:51 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button in toolbar for the "Arbitration" namespace in ruwiki (T213049) (duration: 00m 52s)
  • 12:44 moritzm: installing OpenSSL 1.0.2 security updates for stretch
  • 12:40 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable reader trust survey (T209882) (duration: 01m 07s)
  • 12:02 gehel: repool wdqs100[78] - data import complete - T213210
  • 11:55 jynus: enabling gtid on db1124:s5
  • 11:54 jynus: enabling gtid on db1082
  • 11:23 jynus: stopping db1082 and db2052 s5 replication in sync to migrate db1124:s5 master
  • 10:30 moritzm: fixed package installation status on db2062
  • 10:01 volans: upgraded spicerack to 0.0.11 on cumin2001 T205884
  • 10:00 volans: uploaded spicerack_0.0.11 to apt.wikimedia.org stretch-wikimedia T205884
  • 09:44 hashar: Some CI npm jobs get broken due to a faulty node module. https://phabricator.wikimedia.org/T213249
  • 09:38 banyek: repooling labdsb1010 - T210693
  • 09:26 banyek: dropping materialized views on labdb1010 - T210693
  • 09:26 banyek: depooled labsdb1010
  • 08:28 moritzm: installing openssl security updates for on stretch-based DB servers
  • 07:55 moritzm: installing libseccomp updates from stretch point release
  • 07:43 hashar: contint1001: restarted Zuul to take in account SMTP configuration | https://gerrit.wikimedia.org/r/376739 | T93414
  • 06:03 kartik@deploy1001: Finished deploy [cxserver/deploy@1098942]: Update cxserver to 656c468 (duration: 04m 08s)
  • 05:59 kartik@deploy1001: Started deploy [cxserver/deploy@1098942]: Update cxserver to 656c468
  • 01:15 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: T213227 Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s)
  • 01:04 jforrester@deploy1001: Synchronized docroot/: T187716 Remove mobilelanding.php, no longer pointed to by Apache (duration: 00m 52s)
  • 00:58 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [Wikimania] Add 2019 content to default search (duration: 00m 53s)
  • 00:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202683 [Wikimania] Create year namespaces for each Wikimania, 2005–2019 (duration: 00m 53s)
  • 00:34 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Make password policy and logging code saner (duration: 00m 52s)
  • 00:33 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make password policy and logging code saner (duration: 00m 55s)

2019-01-08

  • 23:44 SMalyshev: repooled wdqs1004
  • 23:35 eileen: process-control config revision is 9dc6e63fcd
  • 23:00 XioNoX: Update pfw3-codfw/eqiad security policies - T213100
  • 22:39 XioNoX: deactivate policy-statement BGP_fundraising_aggregates term nat on pfw3-eqiad/codfw - T211028
  • 22:29 gehel: starting data copy from wdqs1007 to wdqs1008 (both will be depooled) - T213217
  • 22:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Add default search NSes (duration: 00m 51s)
  • 22:22 James_F: Ran /docroot/noc/createTxtFileSymlinks.sh for new dblist
  • 22:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikidatarepo dblist where appropriate (duration: 00m 52s)
  • 22:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: dblists: Load wikibaserepo (duration: 00m 52s)
  • 22:15 jforrester@deploy1001: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 22:14 jforrester@deploy1001: Synchronized dblists/wikidata.dblist: dblists: Remove testcommons from wikidata list (duration: 00m 52s)
  • 22:13 jforrester@deploy1001: Synchronized dblists/wikidatarepo.dblist: dblists: Add wikidatarepo list (duration: 00m 53s)
  • 22:12 urandom: forcing removal of restbase1016-b (host down way too long to salvage) -- T212418
  • 22:08 marostegui: Drop valid_tag table from db2043 with replication (s3 codfw master - lag will be generated) - T212254
  • 22:03 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: cleanup - Idfa129a65a41 (duration: 00m 53s)
  • 21:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 T212254 (duration: 00m 52s)
  • 21:49 marostegui: Drop valid_tag table from db1078 (s3) - T212254
  • 21:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 T212254 (duration: 00m 53s)
  • 21:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 T212254 (duration: 00m 53s)
  • 21:38 marostegui: Drop valid_tag table from db1123 (s3) - T212254
  • 21:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 T212254 (duration: 00m 53s)
  • 21:31 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.12
  • 21:03 dduvall@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache (duration: 39m 22s)
  • 20:42 ejegg: updated payments-wiki from b8acb95a2a to c455bbc6bb
  • 20:24 dduvall@deploy1001: Started scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache
  • 20:24 gehel: starting data copy from wdqs1004 to wdqs1007 (both will be depooled) - T213217
  • 20:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Don't enable entities, we're not Wikidata.org (duration: 01m 44s)
  • 20:11 XioNoX: change BGP_fundraising_aggregates term nat from static to aggregate on pfw3-eqiad - T211028
  • 19:51 ejegg: updated fundraising CiviCRM from b8e3a71845 to 5580f0b11c
  • 19:48 krinkle@deploy1001: Finished deploy [performance/navtiming@68fd54d]: (no justification provided) (duration: 00m 05s)
  • 19:48 krinkle@deploy1001: Started deploy [performance/navtiming@68fd54d]: (no justification provided)
  • 19:48 dduvall@deploy1001: Pruned MediaWiki: 1.33.0-wmf.12 (duration: 06m 26s)
  • 19:11 arlolra: Updated Parsoid to 2c5dc7b (T197616, T205491, T209772, T199926, T209194, T204622)
  • 19:06 marostegui: Drop valid_tag table from s1 - T212254
  • 19:00 arlolra@deploy1001: Finished deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b (duration: 10m 40s)
  • 18:54 XioNoX: make pfw3-codfw source NAT similar to pfw3-eqiad - T211028
  • 18:54 ejegg: updated SmashPig standalone install from fb3268897b to 25713ca232
  • 18:50 marostegui: Drop valid_tag table from s4 - T212254
  • 18:50 XioNoX: add NAT workaround to pfw3-eqiad - T211028
  • 18:49 arlolra@deploy1001: Started deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b
  • 18:38 XioNoX: temporarily permit ssh from frpm1001 to pfw3-eqiad on pfw3-eqiad
  • 18:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T86338 T202167 (duration: 00m 45s)
  • 18:27 jynus: restarting s5 replication on labsdb1009/10/11
  • 17:41 moritzm: installing libseccomp updates from stretch point release
  • 17:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 (duration: 02m 29s)
  • 17:38 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2
  • 17:37 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - T210752 T197616 (duration: 96m 50s)
  • 17:33 _joe_: applying the new apache configuration to jobrunners in eqiad
  • 17:24 elukey: roll restart of aqs on aqs100* to pick up new Druid settings
  • 17:20 _joe_: depooling mw1299 for testing of the apache change
  • 17:16 SMalyshev: restarted Blazegraph wdqs1006 due to unresponsiveness (caused by load?)
  • 16:56 urandom: forcing removal of restbase1016-a (host down way too long to salvage) -- T212418
  • 16:56 jynus: changing db1124:s5 replication to db2066
  • 16:55 marostegui: Deploy schema change on db1105:3311 T86338 T202167
  • 16:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T86338 T202167 (duration: 00m 44s)
  • 16:54 jynus: stopping s5 replication on labsdb1009/10/11 to prevent undoable mistakes
  • 16:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 - T212833 (duration: 02m 51s)
  • 16:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 T86338 T202167 (duration: 00m 45s)
  • 16:12 XioNoX: add BGP sessions to AS64050 in AMS-IX
  • 16:04 marostegui: Drop valid_tag table from s7 - T212254
  • 16:00 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - T210752 T197616
  • 15:59 marostegui: Deploy schema change on db1089 T86338 T202167
  • 15:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 T86338 T202167 (duration: 00m 45s)
  • 15:45 marostegui: Drop valid_tag table from s2 - T212254
  • 15:32 marostegui: Stop MySQL on es2019 for upgrade - T212833
  • 15:23 godog: briefly stop carbon daemons on graphite1004 to move /srv/whisper -> /srv/carbon/whisper
  • 15:17 marostegui: Increase connections from 10 to 50 for recommendationapiservice on m2 - T212154
  • 15:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2019 - T212833 (duration: 00m 44s)
  • 15:04 hashar: Restarted CI Jenkins
  • 13:02 zeljkof: EU SWAT finished
  • 12:59 jynus: transfering db1102:s5 mariadb datadir to db1082
  • 12:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Give all users (including IPs) the pagequality right in plwikisource (T212478) (duration: 00m 45s)
  • 12:45 akosiaris@deploy1001: scap-helm zotero finished
  • 12:45 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 12:45 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 12:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow ptwikis bureaucrats to grant/revoke rollbacker user group (T212735) (duration: 00m 45s)
  • 12:39 akosiaris@deploy1001: scap-helm zotero upgrade production2 -f zoterov2-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use localized wgMetaNamespace and wgMetaNamespaceTalk in satwiki (T211294) (duration: 00m 45s)
  • 12:23 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for students writing Wikipedia program (T212226) (duration: 00m 44s)
  • 12:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for University of Southern California editathon (T212917) (duration: 00m 45s)
  • 12:07 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T212768 [cirrus] re-enable HHVM connection pooling (duration: 00m 45s)
  • 12:01 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource (duration: 12m 38s)
  • 11:49 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource
  • 11:46 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Increase time out on the MW side to 60s - T204183 (duration: 00m 51s)
  • 11:36 akosiaris@deploy1001: scap-helm zotero finished
  • 11:36 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 11:36 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 11:35 akosiaris@deploy1001: scap-helm zotero finished
  • 11:35 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 11:35 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 11:33 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron strugling, restart - T213154
  • 11:29 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw
  • 11:24 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=zotero,name=codfw
  • 11:07 jynus: stoping and restarting db1102 (s5, s4) for upgrade
  • 11:04 moritzm: rebooting mw1261
  • 10:48 moritzm: installing libseccomp updates from stretch point release
  • 10:34 dcausse: elastic@eqiad setting crosscluster conf on production search cluster (T213150)
  • 10:25 banyek: executing schema change on db1062 - T85757
  • 09:39 foks: reset user email for Zergiorubio
  • 09:26 akosiaris@deploy1001: scap-helm zotero finished
  • 09:26 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 09:26 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 09:22 jynus: stop replication on db1124:s5 T213108
  • 09:21 akosiaris@deploy1001: scap-helm zotero finished
  • 09:21 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 09:21 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # T101131
  • 09:12 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - T192691 (duration: 00m 59s)
  • 09:12 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - T192691
  • 05:39 SMalyshev: restarted some Blazegraph servers as precaution against corruption issues
  • 04:26 onimisionipe: depooling wdqs1008 - T213134
  • 03:23 kartik@deploy1001: Finished deploy [cxserver/deploy@b669f95]: Update cxserver to d6b1d6f (duration: 05m 00s)
  • 03:18 kartik@deploy1001: Started deploy [cxserver/deploy@b669f95]: Update cxserver to d6b1d6f
  • 00:22 gehel: restarting tilerator on all maps servers
  • 00:06 gehel: depooling wdqs1007 (something looks like DB corruption)

2019-01-07

  • 23:56 eileen: update civicrm revision changed from bcb4b7a7d1 to b8e3a71845, config revision is 260be32d0a
  • 22:08 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Re-enable uploading of files, accidentally prevented (duration: 00m 44s)
  • 21:19 XioNoX: push NAT changes to pfw3-eqiad - T211028
  • 21:16 awight@deploy1001: Finished deploy [ores/deploy@9253beb]: T212530: new ORES models; revscoring 2.3.0 (duration: 15m 28s)
  • 21:13 mforns@deploy1001: Finished deploy [analytics/refinery@faac592]: deploying analytics/refinery to account with refinery-source v0.0.83 (duration: 06m 52s)
  • 21:06 mforns@deploy1001: Started deploy [analytics/refinery@faac592]: deploying analytics/refinery to account with refinery-source v0.0.83
  • 21:00 awight@deploy1001: Started deploy [ores/deploy@9253beb]: T212530: new ORES models; revscoring 2.3.0
  • 20:19 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Final go-switch for WBMI Ie52b8af006ba (duration: 00m 45s)
  • 19:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove redundant namespace talk definitions (T206952) (duration: 00m 44s)
  • 19:46 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgMetaNamespace for bewikibooks (T212665) (duration: 00m 45s)
  • 19:43 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseRepo and WikibaseMediaInfo on testcommonswiki (duration: 00m 44s)
  • 19:42 XioNoX: push firewall change to pfw3-codfw/eqiad - T211712
  • 19:40 catrope@deploy1001: Synchronized wmf-config/Wikibase.php: Set empty clientDbList for testcommonswiki (duration: 00m 44s)
  • 19:38 catrope@deploy1001: Synchronized dblists/wikidata.dblist: Enable Wikidata on testcommonswiki (duration: 00m 44s)
  • 19:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add importupload to sysops on testcommons (duration: 00m 45s)
  • 19:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on viwikisource (T212929) (duration: 00m 45s)
  • 19:13 catrope@deploy1001: Synchronized dblists/flow.dblist: Enable Flow on viwikisource (T212929) (duration: 00m 45s)
  • 19:11 RoanKattouw: Ran emptyUserGroup.php for autoreview, reviewer and editor groups on srwikinews (T212058)
  • 18:51 XioNoX: re-deactivate bgp sessions to Zayo on cr1-eqiad - T212791
  • 18:20 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@d8f911c]: new GUI, Updater & Blazegraph build (duration: 10m 13s)
  • 18:18 XioNoX: activate bgp sessions to Zayo on cr1-eqiad - T212791
  • 18:10 jynus: manually creating tables on es1015, es1017 with replication for testcommonswiki
  • 18:10 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d8f911c]: new GUI, Updater & Blazegraph build
  • 18:07 onimisionipe@deploy1001: deploy aborted: (no justification provided) (duration: 00m 04s)
  • 18:06 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d8f911c]: (no justification provided)
  • 18:05 XioNoX: deactivate bgp sessions to Zayo on cr1-eqiad T212791
  • 17:35 akosiaris: restart pdfrender on scb1004
  • 17:35 akosiaris: restart pdfrender
  • 17:23 kartik@deploy1001: Finished deploy [cxserver/deploy@594420b]: Update cxserver to 7632c43 (duration: 04m 06s)
  • 17:19 kartik@deploy1001: Started deploy [cxserver/deploy@594420b]: Update cxserver to 7632c43
  • 16:24 jynus: shutting down mariadb again and rebooting db1107
  • 16:15 jynus: starting mariadb on db1107
  • 16:12 onimisionipe: starting inplace reindexing for enwiki - T212224
  • 16:07 volans: powercycle db1107
  • 16:03 elukey: stop eventlogging mysql consumers on eventlog1002 and eventlogging replication on db1108 due to issues with db1107
  • 16:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 00m 45s)
  • 15:46 cmjohnson1: replacing bad fuse on the PDU rack A2 eqiad
  • 14:19 moritzm: added jbond to WMF-LDAP group in Phabricator (T213079)
  • 13:56 ariel@deploy1001: Finished deploy [dumps/dumps@acd9bca]: logging and quiet mode for adds-changes and other dumps (duration: 00m 05s)
  • 13:56 ariel@deploy1001: Started deploy [dumps/dumps@acd9bca]: logging and quiet mode for adds-changes and other dumps
  • 13:02 zeljkof: EU SWAT finished
  • 13:01 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus: increase number of shards (T212224) (duration: 00m 44s)
  • 12:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Restrict moving categories for users at srwiki (T213050) (duration: 00m 44s)
  • 12:40 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Cleanup old throttle rules (duration: 00m 44s)
  • 12:34 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: To lift a cap on account creation from IP for mrwiki community (T212921) (duration: 00m 43s)
  • 12:30 Zoranzoki21: tools.zoranzoki21wiki Archived https://www.mediawiki.org/w/index.php?title=Extension:Woopra (https://www.wikidata.org/wiki/Q21679347) - T212994
  • 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reader trust survey (T209882) (duration: 00m 45s)
  • 12:21 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Quiz extension on ru.wikibooks (T212622) (duration: 00m 45s)
  • 12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add suppressredirect user right to editor user group at pl.wikisource (T212655) (duration: 00m 44s)
  • 12:11 gtirloni: disabled notifications for cloudvirt0124 (T212360)
  • 12:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extendedmover user group at en.wiktionary (T212662) (duration: 00m 46s)
  • 12:07 kartik@deploy1001: Finished deploy [cxserver/deploy@2d54a64]: Deploy Google Translation (T90208) (duration: 05m 07s)
  • 12:02 kartik@deploy1001: Started deploy [cxserver/deploy@2d54a64]: Deploy Google Translation (T90208)
  • 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1079 after schema change - T85757 (duration: 00m 44s)
  • 10:31 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Move group1 to new logging infrastructure - T211124 (duration: 00m 45s)
  • 10:30 banyek: repooling db1079 after schema change - T85757
  • 10:27 banyek: restarting replication on db1079 - T85757
  • 09:55 banyek: executing schema change on db1079 with replication enabled - T85757
  • 09:53 banyek: stopping replication on db1079 - T85757
  • 09:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1079 for schema change - T85757 (duration: 01m 02s)
  • 09:36 banyek: depooling db1079 for schema change - T85757
  • 08:30 moritzm: rolling restart of swift backend servers to pick up OpenSSL security update
  • 07:24 elukey: restart pdfrender on scb1002

2019-01-06

  • 14:50 ariel@deploy1001: Finished deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag (duration: 00m 06s)
  • 14:50 ariel@deploy1001: Started deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag

2019-01-05

  • 20:23 elukey: manually clean up of big logs under /var/log/.. on analytics-tool1002 due to root partition almost filled up

2019-01-04

  • 23:07 mutante: scandium apt-get remove nodejs nodes-legacy ; puppet agent -tv - after merging gerrit:482150 this fixed "you have held broken packages" issue, now we are at a puppet dependecy cycle with apt::pin T201366
  • 15:42 bawolff@deploy1001: Synchronized private/PrivateSettings.php: T212667 - More aggressive anti-spam measures for account creation on kowiki (duration: 00m 48s)
  • 14:08 moritzm: rebooting etcd1001-1003 to pick up SSBD-enabled qemu
  • 13:52 moritzm: rebooting etcd1004-1006 to pick up SSBD-enabled qemu
  • 13:33 moritzm: rebooting kubernetes staging etcd hosts to pick up SSBD-enabled qemu
  • 13:11 moritzm: rebooting kubernetes staging master to pick up SSBD-enabled qemu
  • 12:57 moritzm: rebooting kubernetes staging workers for kernel security update
  • 11:58 moritzm: installing libsndfile security updates
  • 11:33 moritzm: installing jasper security updates
  • 11:31 moritzm: installing libdatetime-timezone-perl updates for recent tz changes
  • 10:47 arturo: T212898 reimaging cloudvirt1024 as stretch
  • 10:46 moritzm: rolling restart of swift proxies to pick up OpenSSL update
  • 09:57 jijiki: restarting thumbor services to pick up 481141
  • 09:50 onimisionipe: restarting nginx on all wdqs hosts
  • 09:40 banyek: executing schema change on dbstore1002 - T85757
  • 09:13 moritzm: restarting nginx on puppetdb hosts to pick up new OpenSSL
  • 09:03 banyek: executing schema change on db1116 - T85757
  • 08:44 moritzm: restarting nginx on francium to pick up new OpenSSL
  • 08:16 elukey: restart eventlogging daemons on eventlog1002 to pick up openssl updates
  • 07:56 moritzm: installing OpenSSL security updates
  • 00:07 mutante: an-coord1001 - apt-get clean to free disk space, reacting to Icinga alert for running out of disk

2019-01-03

  • 23:08 volans: restarted pdfrender on scb1004
  • 22:29 volans: restarted all slaves on dbstore1002 (relayed from banyek)
  • 22:14 banyek: stopping all slaves on dbstore1002 (NOT labsdb)
  • 22:14 banyek: stopping all slaves on labsdb1002
  • 20:50 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: Fix error for testcommons (duration: 00m 44s)
  • 20:46 reedy@deploy1001: Synchronized dblists/group0.dblist: Add testcommonswiki to group0 (duration: 00m 43s)
  • 20:43 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 05s)
  • 20:24 reedy@deploy1001: Synchronized wmf-config/db-codfw.php: T197616 (duration: 00m 44s)
  • 20:23 reedy@deploy1001: Synchronized wmf-config/db-eqiad.php: T197616 (duration: 00m 44s)
  • 20:13 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197616 (duration: 00m 44s)
  • 20:12 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: T197616 (duration: 00m 44s)
  • 20:11 reedy@deploy1001: rebuilt and synchronized wikiversions files: T197616
  • 20:09 reedy@deploy1001: Synchronized dblists/: T197616 (duration: 00m 45s)
  • 18:51 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@1182b3b]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests, part 2 (duration: 05m 27s)
  • 18:46 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@1182b3b]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests, part 2
  • 18:37 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@c470ed2]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests (duration: 04m 11s)
  • 18:33 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@c470ed2]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests
  • 18:21 volans: restart pdfrender on scb1003
  • 17:58 ariel@deploy1001: Finished deploy [dumps/dumps@10dc8ad]: return properly if commands failed (duration: 00m 08s)
  • 17:58 ariel@deploy1001: Started deploy [dumps/dumps@10dc8ad]: return properly if commands failed
  • 16:32 XioNoX: remove old 10.64.22.0/24 IPs from cloud-instance-transport1-b-eqiad - T207663
  • 16:22 moritzm: rebooting kubernetes workers in eqiad for kernel security update
  • 16:02 arturo: reimaging cloudvirt1013 cloudvirt1026-1028 to stretch
  • 15:48 moritzm: restart parsoid on wtp1025 to pick up OpenSSL update for nodejs
  • 15:43 jijiki: Enabled puppet on mw servers after merging 481796 - T197616
  • 15:31 jijiki: Disabling puppet on mw servers to test 481796 - T197616
  • 15:14 ejegg: updated Fundraising CiviCRM from b33dcd3c94 to bcb4b7a7d1
  • 14:37 moritzm: rebooting kubernetes workers in codfw for kernel security update
  • 14:37 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1101:3317 after schema change - T85757 (duration: 00m 44s)
  • 14:32 banyek: repooling db1101:3317 after schema change - T85757
  • 14:21 moritzm: rebooting kubernetes masters in eqiad to pick up SSBD-enabled qemu
  • 14:14 moritzm: rebooting kubernetes mastes in codfw to pick up SSBD-enabled qemu
  • 14:05 arturo: T209616 reimage cloudvirt1029 as debian stretch
  • 13:43 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1101:3317 for schema change - T85757 (duration: 00m 44s)
  • 13:41 banyek: depooling db1101:3317 for schema change - T85757
  • 13:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1098:3317 after schema change - T85757 (duration: 00m 44s)
  • 13:34 banyek: repooling db1098:3317 after schema change - T85757
  • 13:24 kartik@deploy1001: Finished deploy [cxserver/deploy@3b2ede7]: Update cxserver to 2369a18 (duration: 04m 30s)
  • 13:20 kartik@deploy1001: Started deploy [cxserver/deploy@3b2ede7]: Update cxserver to 2369a18
  • 12:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1098:3317 for schema change - T85757 (duration: 00m 45s)
  • 12:55 banyek: depooling db1098:3317 for schema change - T85757
  • 12:54 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1094 after schema change - T85757 (duration: 00m 45s)
  • 12:49 banyek: repooling db1094 after schema change - T85757
  • 12:41 arturo: T212302 reimaging again cloudvirt1030 to test final puppet code
  • 12:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1094 for schema change - T85757 (duration: 00m 46s)
  • 12:28 banyek: depooling db1094 for schema change - T85757
  • 12:27 moritzm: restarting tor on torrelay1001 to pick up OpenSSL security update
  • 11:02 _joe_: manually reloading icinga to pick up changes to commands.cfg
  • 10:55 moritzm: installing apache updates on puppetmasters
  • 10:22 moritzm: installing ghostscript security updates on jessie
  • 09:51 elukey: restart memcached on mc1023 to apply -R 200 - T208844
  • 09:46 moritzm: remove imagemagick remnants from ATS hosts (obsoleted by upstream packaging change which dropped the webp plugin)
  • 09:39 moritzm: installing nginx updates on puppetdb*
  • 09:26 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2019 - T212833 (duration: 01m 33s)
  • 09:18 banyek: repooling es2019 - T212833
  • 08:46 moritzm: rolling restart of proton to pick up OpenSSL update
  • 08:35 banyek: depooled es2019 as host was unsresponsive - T212833
  • 08:35 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2019, host is unsresponsible - T212833 (duration: 00m 49s)
  • 08:11 moritzm: installing OpenSSL security updates
  • 00:21 mutante: notebook1004 - started nagios-nrpe-server one more time

2019-01-02

  • 23:59 mutante: notebook1004 still keeps running out of memory from some user actions and that kills nagios-nrpe-server and that causes a bunch of Icinga alerts
  • 23:39 mutante: notebook1004 - systemctl start nagios-nrpe-server
  • 23:39 mutante: notebook1004 - systemctl status nagios-nrpe-server
  • 20:59 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=parsoid,service=parsoid,name=wtp1028.eqiad.wmnet
  • 20:59 herron: repooling wtp1028 T212624
  • 20:52 herron: rebooting wtp1028 — looking for POST errors T212624
  • 20:05 Krinkle: mwmaint1002: foreachwikiindblist s5 deleteEqualMessages.php
  • 20:04 Krinkle: mwmaint1002: foreachwikiindblist s2 deleteEqualMessages.php
  • 18:35 volans: restarting icinga on icinga1001 T212669
  • 16:50 XioNoX: create BGP sessions to AS3214 in AMS-IX
  • 16:46 XioNoX: remove BGP sessions to AS42949 in AMS-IX (leaving the IX)
  • 16:43 XioNoX: remove BGP sessions to AS6866 in AMS-IX (leaving the IX)
  • 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1090:3317 after schema change - T85757 (duration: 00m 46s)
  • 16:30 arturo: reimaging cloudvirt1030 with stretch, server cleanup after puppet refactoring
  • 16:29 moritzm: restarting Superset to pick up openssl security update
  • 16:25 moritzm: restarting Hue to pick up openssl security update
  • 16:23 arturo: T212302 re-enable puppet in all {cloud,lab}virt* servers, all was fine
  • 16:22 banyek: repooling db1090:3317 after schema change (T85757)
  • 16:11 arturo: T212302 disable puppet in all {cloud,lab}virt* servers to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/481194/
  • 15:39 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1090:3317 for schema change - T85757 (duration: 00m 44s)
  • 15:34 moritzm: installing OpenSSL security updates
  • 15:31 banyek: depooling db1090:3317 for schema change (T85757)
  • 15:13 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1086 after schema change - T85757 (duration: 00m 44s)
  • 15:07 banyek: repooling db1086 after schema change (T85757)
  • 14:49 banyek: executing schema change on db1086 - T85757
  • 14:48 moritzm: installing ghostscript security update for jessie
  • 14:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1086 for schema change - T85757 (duration: 00m 45s)
  • 14:38 banyek: depooling db1086 for schema change (T85757)
  • 14:15 ema: cp hosts: upgrade OpenSSL from 1.1.0f to 1.1.0j
  • 13:39 moritzm: installing ghostscript update for stretch
  • 13:33 moritzm: installing libav security updates
  • 13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 T86338 T202167 (duration: 00m 44s)
  • 13:17 moritzm: installing openjpeg2 security updates
  • 13:17 banyek: executing schema change on db2040 (s7 codfw master) replication lag could be expected on codfw - T85757
  • 13:13 banyek: stopping replication on db2077 prior to executing schema change on codfw s7 master (db2040) - T85757
  • 13:06 marostegui: Deploy schema change on db1119 - T86338 T202167
  • 13:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 T86338 T202167 (duration: 00m 45s)
  • 13:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T86338 T202167 (duration: 00m 47s)
  • 12:00 moritzm: rebooting labtestpuppetmaster2001 for kernel security update
  • 11:53 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe1006.eqiad.wmnet
  • 11:51 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.eqiad.wmnet
  • 11:50 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.codfw.wmnet
  • 11:46 ema: replace TLS certificates on ms-fe eqiad hosts T212215
  • 11:41 moritzm: rebooting labtestweb2001 for kernel security update
  • 11:24 marostegui: Deploy schema change on db1099:3311 - T86338 T202167
  • 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T86338 T202167 (duration: 00m 45s)
  • 11:17 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
  • 11:10 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
  • 10:59 ema: replace TLS certificates on ms-fe codfw hosts T212215
  • 10:52 moritzm: rebooting centrallog1001 for kernel security update
  • 10:48 volans: testing the new spicerack package on cumin2001, in the unlikely event you need to use spicerack cookbooks today please use cumin1001
  • 10:45 godog: ms-be2018 Flashing Smart Array P840 in Slot 3 [ 3.00 -> 6.60 ]
  • 10:43 moritzm: removed labvirt1013 from debmonitor, got renamed in T212513
  • 10:42 volans: uploaded spicerack_0.0.10-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 10:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2096 (duration: 00m 44s)
  • 09:50 marostegui: Stop MySQL on db2096 for kernel and mysql upgrade
  • 09:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2096 (duration: 00m 45s)
  • 09:48 marostegui@deploy1001: sync-file aborted: Depool db2096 (duration: 00m 01s)
  • 09:18 moritzm: installing c3p0 security updates
  • 09:07 Zoranzoki21: Drop valid_tag from s8 by Marostegui - T212254
  • 09:06 godog: eqiad-prod: final weight for ms-be10[44-50].eqiad.wmnet - T209618
  • 08:56 moritzm: installing libarchive security updates
  • 07:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T212692 (duration: 00m 46s)
  • 07:30 marostegui: Fix login.logging table on db1078 - T212692
  • 07:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T212692 (duration: 00m 47s)
  • 07:01 marostegui: Deploy schema change on s1 codfw master (lag will be generated on s1 codfw) - T202167 T86338
  • 06:54 marostegui: Drop empty valid_tag table from labswiki labtestwiki - T212254
  • 06:49 marostegui: Drop empty valid_tag table from s5 - T212254
  • 06:25 marostegui: Drop valid_tag from s6 - T212254
  • 06:15 marostegui: Fix last chunks on db1124:338 - T212574


Archives

See Server admin log/Archives.