You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(hoo: Updated operations/dumps/dcat (559dee37452..a86285f4e7) on snapshot1008)
imported>Stashbot
(elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred.")
Line 1: Line 1:
== 2019-01-13 ==
== 2019-02-03 ==
* 16:33 hoo: Updated operations/dumps/dcat (559dee37452..a86285f4e7) on snapshot1008
* 01:10 elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred."


== 2019-01-12 ==
== 2019-02-02 ==
* 21:46 akosiaris: restart all zotero pods in eqiad
* 20:42 chaomodus: restarted pdfrender on scb1003
* 16:12 moritzm: rebooting mw2167 for a test
* 20:41 chaomodus: restarted pdfrender on scb1004
* 02:16 legoktm@deploy1001: Synchronized docroot/mediawiki.org/keys: Add Mukunda's new subkey that was used for the 1.32 release - [[phab:T213521|T213521]] (duration: 00m 47s)
* 20:06 chaomodus: parsoid was failed on scandium and alerting, the service parsoid-vd was restarted and appears to have come back
* 05:44 jforrester@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/VisualEditor/lib/ve/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js: b/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js [[phab:T214963|T214963]] Hot-deploy VE fix to stop hitting user pref writes without debounce (duration: 01m 02s)


== 2019-01-11 ==
== 2019-02-01 ==
* 21:56 jforrester@deploy1001: Finished scap: Full scap sync to update wmf.12 i18n for the weekend {{Gerrit|Idf2a67860f}} (duration: 19m 12s)
* 23:16 vgutierrez: restart pdfrender on scb1004
* 21:37 jforrester@deploy1001: Started scap: Full scap sync to update wmf.12 i18n for the weekend {{Gerrit|Idf2a67860f}}
* 21:57 ejegg: updated payments-wiki-staging from {{Gerrit|7767c7027e}} to {{Gerrit|52a271e681}}
* 18:43 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: Update ExtensionDistributor for 1.32 release - https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/483735 (duration: 00m 46s)
* 21:25 ejegg: updated payments-wiki-staging to fundraising/REL1_31 branch
* 18:07 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2060 [[phab:T210713|T210713]] (duration: 00m 46s)
* 07:13 bawolff_: reset 2FA on wikitech for [[User:Cicalese]]
* 17:10 marostegui: Deploy schema change on db2060 - [[phab:T210713|T210713]]
* 16:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2060 [[phab:T210713|T210713]] (duration: 00m 46s)
* 16:53 marostegui: Defragment change_tag table on db2060 - [[phab:T210713|T210713]]
* 14:37 jynus: upgrade and restart db2091 (s2, s4)
* 14:12 jynus: updating mariadb client packages on cumin* hosts
* 11:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: repool es1018 fully (duration: 00m 46s)
* 11:21 jynus: stop, upgrade and reboot es2017
* 11:04 jynus: stop, upgrade and reboot es2016
* 10:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: repool es1018 with low load (duration: 00m 46s)
* 10:31 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2013 (duration: 00m 45s)
* 10:30 jynus: upgrade and restart es1018
* 09:58 jynus: upgrade and reboot es2013
* 09:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2013 (duration: 00m 45s)
* 09:49 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2013 (duration: 00m 47s)
* 09:32 jynus: reset iLo on db2053
* 08:49 moritzm: installing tmpreaper security updates
* 02:40 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|Ib87407165382}} (duration: 00m 46s)
* 01:20 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT [[phab:T211993|T211993]] Enable GrowthExperiments help panel for 50% of new users on cswiki and kowiki (duration: 00m 46s)
* 01:05 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT [[phab:T211993|T211993]] Enable GrowthExperiments help panel on cswiki and kowiki (duration: 00m 45s)
* 01:03 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/WikimediaEvents/includes/PageViews.php: SWAT: [[phab:T213186|T213186]] GrowthExperiments: Support templates for help desk title (duration: 00m 46s)
* 00:50 XioNoX: bump prefix limit for AS6939 in eqsin
* 00:18 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/AbuseFilter/includes/AbuseFilterHooks.php: [[phab:T213453|T213453]]: Use slot in onEditFilterMergedContent and newVariableHolderForEdit in AbuseFilter (duration: 00m 47s)
* 00:12 James_F: 482373 is live on mwdebug1002 for extensive checks.
* 00:08 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Help panel: Set help desk page correctly on kowiki {{Gerrit|Ia94cfc571}} (duration: 00m 46s)
 
== 2019-01-10 ==
* 23:45 Krinkle: krinkle@tungsten: upgrade xhgui to include upstream {{Gerrit|f039fb9f99f}} - [[phab:T213218|T213218]]
* 23:45 Krinkle: upgraded xhgui to upstream {{Gerrit|2965240c91e52}} (current upstream master) - [[phab:T213218|T213218]]
* 23:36 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: [[phab:T213497|T213497]] [Commons, TestCommons] Don't use Wikibase entity search (duration: 00m 46s)
* 22:57 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/includes/EditEntity/MediawikiEditFilterHookRunner.php: [[phab:T213453|T213453]]: Pass slotrole into EditFilterMergedContent hook in Wikibase repo (duration: 00m 47s)
* 20:47 marxarelli: both mediawiki error rates and 500 response rates have subsided back to pre-deploy levels
* 20:19 marxarelli: seeing increase in "60 second timed out" error rate and rise in 503 rate, as was the case with group1 deployment. continuing to monitor
* 20:11 gehel: restart blazegraph on wdqs1009 to validate new config
* 20:02 tgr@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/WikimediaEvents/modules/ve-wme/campaigns.js: SWAT: [[gerrit:483485{{!}}Remove unnecessary addPlugin wrapper (T213338)]] (duration: 00m 53s)
* 19:50 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481987{{!}}Remove AICaptcha settings (T186244)]] (duration: 00m 52s)
* 19:47 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:468831{{!}}Whitelist *.*.archive.org in wgCopyUploadsDomains (T207581)]] (duration: 00m 53s)
* 19:41 tgr: ran mwscript namespaceDupes.php bnwikibooks --fix (238 links fixed)
* 19:41 volans: installed spicerack 0.0.12-1 on cumin2001 [[phab:T205884|T205884]]
* 19:39 volans: uploaded spicerack_0.0.12-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [[phab:T205884|T205884]]
* 19:39 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:458870{{!}}Note that namespaceDupes.php maintenance script run will be needed after the deployment. (T203534)]] (duration: 00m 53s)
* 19:14 marostegui: Deploy schema change on dbstore1001 - [[phab:T85757|T85757]]
* 19:13 marostegui: Deploy schema change on dbstore1002 - [[phab:T85757|T85757]]
* 18:57 tzatziki: deleting three files for legal compliance
* 18:52 anomie@mwmaint1002: Running migrateActors.php on test wikis and mediawikiwiki for [[phab:T188327|T188327]]. This may cause lag in codfw.
* 18:47 marostegui: Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - [[phab:T85757|T85757]]
* 18:46 marostegui: Stop replication on s1 codfw master for a schema change - [[phab:T85757|T85757]]
* 18:37 marostegui: Stop replication on s8 codfw master for a schema change - [[phab:T85757|T85757]]
* 18:30 marostegui: Upgrade mysql and kernel on db2060
* 18:30 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2053, db2060 for kernel and mysql upgrade (duration: 00m 51s)
* 18:13 marostegui: Stop MySQL on db2046 for kernel upgrade
* 18:12 marostegui: The above change was db2053 and not db2060
* 18:11 marostegui: Stop MySQL on db2053 and db2060 for mysql and kernel upgrade
* 18:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2053, db2060 for kernel and mysql upgrade (duration: 00m 53s)
* 17:50 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2015 (duration: 00m 53s)
* 17:49 marostegui: Deploy schema change on db2053 - [[phab:T210713|T210713]]
* 17:33 marostegui: Deploy schema change on db2046 - [[phab:T210713|T210713]]
* 16:59 jynus: stop and upgrade es2015
* 16:52 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2015 (duration: 00m 52s)
* 16:41 onimisionipe: data transfer from wdqs1004 -> wdqs1006 completed! - [[phab:T213361|T213361]]
* 16:32 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T159708|T159708]] Enable Structured Data on Commons, captions-only (duration: 00m 53s)
* 16:17 James_F: [[phab:T180981|T180981]] Placed patch to enable WBMI on Commons on mwdebug1002
* 16:13 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T180981|T180981]] Add Commons to wikis with WikibaseMediaInfo installed (duration: 00m 52s)
* 16:11 jforrester@deploy1001: Synchronized dblists/wikidatarepo.dblist: [[phab:T180981|T180981]] Add Commons to wikis with WikibaseRepo installed (duration: 00m 54s)
* 16:04 James_F: [[phab:T180981|T180981]] Placed patch to install but not enable WBMI on Commons on mwdebug1002
* 15:56 marostegui: Deploy schema change on db1068 (s4 master) - [[phab:T86338|T86338]]
* 15:31 fsero: rollbacking last zotero codfw deployment
* 15:27 marostegui: Deploy schema change on db1067 (s1 master) - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 15:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 49s)
* 15:24 addshore: [[phab:T208330|T208330]], MariaDB [testcommonswiki]> TRUNCATE TABLE wb_terms; # Was https://phabricator.wikimedia.org/P7973
* 15:22 fsero@deploy1001: scap-helm zotero upgrade production -f /srv/scap-helm/zotero/zotero-values-codfw.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: codfw]
* 15:21 fsero@deploy1001: scap-helm zotero upgrade -f /srv/scap-helm/zotero/zotero-values-codfw.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: codfw]
* 15:20 addshore@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/includes/Content: [[gerrit:483388{{!}}T208330 dont write to wb_terms for mediainfo]] (duration: 00m 54s)
* 15:12 addshore@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Wikibase/repo/includes/Content: [[gerrit:483388{{!}}T208330 dont write to wb_terms for mediainfo]] (duration: 00m 55s)
* 14:59 marostegui: Deploy schema change on db1080 - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 14:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 52s)
* 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 52s)
* 14:42 fsero@deploy1001: scap-helm zotero finished
* 14:42 fsero@deploy1001: scap-helm zotero cluster staging completed
* 14:42 fsero@deploy1001: scap-helm zotero upgrade staging -f /srv/scap-helm/zotero/zotero-values-staging.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: staging]
* 14:36 fsero@deploy1001: scap-helm zotero finished
* 14:36 fsero@deploy1001: scap-helm zotero cluster staging completed
* 14:36 fsero@deploy1001: scap-helm zotero upgrade staging -f /srv/scap-helm/zotero/zotero-values-staging.yaml /srv/deployment-charts/charts/zotero-0.0.1.tgz [namespace: zotero, clusters: staging]
* 14:35 fsero@deploy1001: scap-helm zotero upgrade staging -f /srv/scap-helm/zotero/zotero-values-staging.yaml [namespace: zotero, clusters: staging]
* 14:33 fsero@deploy1001: scap-helm -h finished
* 14:33 fsero@deploy1001: scap-helm -h cluster staging completed
* 14:33 fsero@deploy1001: scap-helm -h  [namespace: -h, clusters: staging]
* 14:33 marostegui: Deploy schema change on db1114 - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 14:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 53s)
* 14:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: depool es1019 (duration: 00m 53s)
* 13:51 arturo: [[phab:T212302|T212302]] icinga downtime for 2h cloudvirt[1013,1024,1026-1030].eqiad.wmnet bc wrong puppet code
* 13:24 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: depool es1018 (duration: 00m 52s)
* 13:10 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2012 (duration: 00m 52s)
* 13:01 zeljkof: EU SWAT finished
* 13:01 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[gerrit:483118{{!}}Remove main page special casing from ruwikibooks and ruwikiquote (T212849)]] (duration: 00m 52s)
* 12:58 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[gerrit:483117{{!}}Remove main page special casing from eswiki (T212849)]] (duration: 00m 53s)
* 12:53 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: [[gerrit:482516{{!}}Turn off main page special casing for svwiki (T213018)]] (duration: 00m 52s)
* 12:46 zfilipin@deploy1001: Synchronized dblists/flow.dblist: SWAT: [[gerrit:478464{{!}}Disable unused Flow extension on ur.wikibooks (T207627)]] (duration: 00m 55s)
* 12:42 onimisionipe: starting data transfer from wdqs1004 -> wdqs1006 - [[phab:T213361|T213361]]
* 12:34 onimisionipe: starting data transfer from wdqs1003 -> wdqs1006 - [[phab:T213361|T213361]] - aborted (nodes are in different cluster)
* 12:28 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:483393{{!}}Re-enable QuickSurveys extension on enwiki (T209882)]] (duration: 00m 52s)
* 12:20 jynus: stop and upgrade es2012
* 12:12 zfilipin@deploy1001: Synchronized dblists/flow.dblist: SWAT: [[gerrit:483111{{!}}Reverted "Revert "Disable unused Flow extension on de.wikiversity"" (T207626)]] (duration: 00m 53s)
* 12:01 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2012 (duration: 00m 52s)
* 11:54 onimisionipe: starting data transfer from wdqs1003 -> wdqs1006 - [[phab:T213361|T213361]]
* 10:59 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T209857|T209857]] Increase CPU benchmark sampling rate (duration: 00m 53s)
* 10:58 fsero: uploaded docker-registry_2.7.0~rc0~wmf1-1 debian package to reprepro for stretch-wikimedia (done yesterday at 17:21 UTC forgot about the log)
* 10:26 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T209857|T209857]] Run CPU benchmark for a portion of navtiming pageloads (duration: 00m 52s)
* 10:10 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T209857|T209857]] Run CPU benchmark for a portion of navtiming pageloads (duration: 00m 53s)
* 09:52 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T187299|T187299]] Decrease ruwiki navtiming rate (duration: 00m 52s)
* 09:45 gilles@deploy1001: Synchronized tests/InitialiseSettingsTest.php: [[phab:T211395|T211395]] [[phab:T211529|T211529]] tests: Assert that extra namespaces have correspondent talk namespaces (duration: 00m 56s)
* 09:34 moritzm: updated thirdparty/php72 component for stretch-wikimedia to 7.2.13
* 01:41 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make GrowthExperiments config wmf.12-proof (duration: 00m 52s)
* 01:21 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Revert latest config patch (caused fatal errors on kowiki) (duration: 00m 52s)
* 00:58 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure help desk page for help panel correctly on kowiki ([[phab:T213186|T213186]]) (duration: 00m 53s)
* 00:56 cstone: updated fundraising tools from {{Gerrit|5f44d9dd43}} to {{Gerrit|da82ed111d}}
* 00:34 catrope@deploy1001: Synchronized php-1.33.0-wmf.12/includes/MovePage.php: Fix missing ATOMIC_CANCELABLE in MovePage::move() ([[phab:T213168|T213168]]) (duration: 00m 53s)
* 00:20 catrope@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/GrowthExperiments/: Help panel fixes ([[phab:T212973|T212973]], [[phab:T212890|T212890]], [[phab:T213186|T213186]]) (duration: 00m 54s)
* 00:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable EventLogging for GrowthExperiments help panel ([[phab:T211991|T211991]]) (duration: 00m 54s)
 
== 2019-01-09 ==
* 23:51 mutante: thumb1004 - still needs broken RAM replaced, expired downtime, re-ACKed ([[phab:T207721|T207721]])
* 23:39 mutante: mw2151 - change netbox status from active to staged - it's not actually active, it's role(spare) and was jessie  ([[phab:T192457|T192457]])
* 23:34 mutante: reinstalling mw2151.codfw.wmnet because it was the very last mw* host on jessie
* 21:20 bblack: multatuli (ns2) - upgrade gdnsd to 9949 beta release
* 21:04 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@bfa9241]: Increase concurrency for categoryMembershipJob [[phab:T192691|T192691]] (duration: 00m 45s)
* 21:04 James_F: Creating Wikibase repo tables on Commons for [[phab:T68108|T68108]]
* 21:03 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@bfa9241]: Increase concurrency for categoryMembershipJob [[phab:T192691|T192691]]
* 21:00 James_F: Running rebuildall on TestCommons
* 20:53 bblack: authdns1001 (ns0) - upgrade gdnsd to 9949 beta release
* 20:45 James_F: Created Wikibase repo tables on TestCommons
* 20:11 dduvall@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.12 (duration: 00m 53s)
* 20:10 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.12
* 19:28 crusnov@deploy1001: Finished deploy [netbox/deploy@7fe39e1]: Deploy Django security upgrade (duration: 04m 33s)
* 19:23 crusnov@deploy1001: Started deploy [netbox/deploy@7fe39e1]: Deploy Django security upgrade
* 19:01 ejegg: updated standalone SmashPig deploy from {{Gerrit|25713ca232}} to {{Gerrit|78b92b7fef}}
* 18:43 bblack: authdns2001 (ns1) - upgrade gdnsd to 9949 beta release
* 18:26 XioNoX: add bgp sessions to AS31800 on cr1-eqsin
* 18:19 marostegui: Rename table tag_summary on enwiki on db1089 - [[phab:T212255|T212255]]
* 18:18 XioNoX: add bgp sessions to AS38895 on cr1-eqsin
* 18:04 marostegui: Drop valid_tag from s3 master (db1075) - [[phab:T212254|T212254]]
* 17:39 tarrow: That last one was SWAT: [[gerrit:483185{{!}}T209504 Increase PHP constraint check entities to 150]]
* 17:36 tarrow@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 53s)
* 17:28 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - [[phab:T86338|T86338]]
* 17:18 James_F: Ran `namespaceDupes.php --wiki=bewikibooks` on mwmaint1002, no change
* 17:16 bblack: uploaded gdnsd-2.99.9949-beta-1+wmf1 to reprepro for stretch-wikimedia
* 17:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 52s)
* 16:29 marostegui: Deploy schema change on db1083 - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 16:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 53s)
* 16:17 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 with full weight (duration: 00m 53s)
* 16:11 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: [[phab:T213227|T213227]] RepoHooks::onApiCheckCanExecute: Only fail if the edit is for our entity's slot (duration: 00m 54s)
* 15:50 marostegui: Drop valid_tag tables from db1095 (s3) - [[phab:T212254|T212254]]
* 15:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 15:23 jijiki: restarting scb* pdfrender
* 15:10 marostegui: Deploy schema change on db1106 (sanitarium s1 master) with replication, lag will be generated on s1 labs - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 15:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 52s)
* 14:39 elukey: restart Hadoop HDFS namenodes on an-master100[1,2] to complete decom of analytics1028->41
* 14:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 [[phab:T212254|T212254]] (duration: 00m 53s)
* 14:36 volans@deploy1001: Finished deploy [debmonitor/deploy@0f096de]: Deploy Django security upgrade (duration: 01m 50s)
* 14:34 volans@deploy1001: Started deploy [debmonitor/deploy@0f096de]: Deploy Django security upgrade
* 14:28 marostegui: valid_tag table on db1077 with replication (lag will be generated on labs s3) - [[phab:T212254|T212254]]
* 14:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 [[phab:T212254|T212254]] (duration: 00m 52s)
* 13:32 urandom: forcing removal of restbase1016-c (host down way too long to salvage) -- [[phab:T212418|T212418]]
* 13:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight (duration: 00m 52s)
* 13:26 zeljkof: EU SWAT finished
* 13:22 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/: SWAT: [[gerrit:483099{{!}}Fix order of arguments in ChangeTags::getPrevTags ([T212703])]] (duration: 05m 50s)
* 13:08 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.12/: SWAT: [[gerrit:483097{{!}}Fix order of arguments in ChangeTags::getPrevTags ([T212703])]] (duration: 06m 54s)
* 13:00 zeljkof: extending eu swat for 5-10 minutes
* 12:51 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:482591{{!}}Enable signature button in toolbar for the "Arbitration" namespace in ruwiki (T213049)]] (duration: 00m 52s)
* 12:44 moritzm: installing OpenSSL 1.0.2 security updates for stretch
* 12:40 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476370{{!}}Disable reader trust survey (T209882)]] (duration: 01m 07s)
* 12:02 gehel: repool wdqs100[78] - data import complete - [[phab:T213210|T213210]]
* 11:55 jynus: enabling gtid on db1124:s5
* 11:54 jynus: enabling gtid on db1082
* 11:23 jynus: stopping db1082 and db2052 s5 replication in sync to migrate db1124:s5 master
* 10:30 moritzm: fixed package installation status on db2062
* 10:01 volans: upgraded spicerack to 0.0.11 on cumin2001 [[phab:T205884|T205884]]
* 10:00 volans: uploaded spicerack_0.0.11 to apt.wikimedia.org stretch-wikimedia [[phab:T205884|T205884]]
* 09:44 hashar: Some CI npm jobs get broken due to a faulty node module. https://phabricator.wikimedia.org/T213249
* 09:38 banyek: repooling labdsb1010 - [[phab:T210693|T210693]]
* 09:26 banyek: dropping materialized views on labdb1010 - [[phab:T210693|T210693]]
* 09:26 banyek: depooled labsdb1010
* 08:28 moritzm: installing openssl security updates for on stretch-based DB servers
* 07:55 moritzm: installing libseccomp updates from stretch point release
* 07:43 hashar: contint1001: restarted Zuul to take in account SMTP configuration {{!}} https://gerrit.wikimedia.org/r/376739 {{!}} [[phab:T93414|T93414]]
* 06:03 kartik@deploy1001: Finished deploy [cxserver/deploy@1098942]: Update cxserver to {{Gerrit|656c468}} (duration: 04m 08s)
* 05:59 kartik@deploy1001: Started deploy [cxserver/deploy@1098942]: Update cxserver to {{Gerrit|656c468}}
* 01:15 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: [[phab:T213227|T213227]] Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s)
* 01:04 jforrester@deploy1001: Synchronized docroot/: [[phab:T187716|T187716]] Remove mobilelanding.php, no longer pointed to by Apache (duration: 00m 52s)
* 00:58 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [Wikimania] Add 2019 content to default search (duration: 00m 53s)
* 00:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T202683|T202683]] [Wikimania] Create year namespaces for each Wikimania, 2005–2019 (duration: 00m 53s)
* 00:34 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:481115{{!}}Make password policy and logging code saner]] (duration: 00m 52s)
* 00:33 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481115{{!}}Make password policy and logging code saner]] (duration: 00m 55s)
 
== 2019-01-08 ==
* 23:44 SMalyshev: repooled wdqs1004
* 23:35 eileen: process-control config revision is {{Gerrit|9dc6e63fcd}}
* 23:00 XioNoX: Update pfw3-codfw/eqiad security policies - [[phab:T213100|T213100]]
* 22:39 XioNoX: deactivate policy-statement BGP_fundraising_aggregates term nat on pfw3-eqiad/codfw - [[phab:T211028|T211028]]
* 22:29 gehel: starting data copy from wdqs1007 to wdqs1008 (both will be depooled) - [[phab:T213217|T213217]]
* 22:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Add default search NSes (duration: 00m 51s)
* 22:22 James_F: Ran /docroot/noc/createTxtFileSymlinks.sh for new dblist
* 22:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikidatarepo dblist where appropriate (duration: 00m 52s)
* 22:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: dblists: Load wikibaserepo (duration: 00m 52s)
* 22:15 jforrester@deploy1001: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
* 22:14 jforrester@deploy1001: Synchronized dblists/wikidata.dblist: dblists: Remove testcommons from wikidata list (duration: 00m 52s)
* 22:13 jforrester@deploy1001: Synchronized dblists/wikidatarepo.dblist: dblists: Add wikidatarepo list (duration: 00m 53s)
* 22:12 urandom: forcing removal of restbase1016-b (host down way too long to salvage) -- [[phab:T212418|T212418]]
* 22:08 marostegui: Drop valid_tag table from db2043 with replication (s3 codfw master - lag will be generated) - [[phab:T212254|T212254]]
* 22:03 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: cleanup - {{Gerrit|Idfa129a65a41}} (duration: 00m 53s)
* 21:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 [[phab:T212254|T212254]] (duration: 00m 52s)
* 21:49 marostegui: Drop valid_tag table from db1078 (s3) - [[phab:T212254|T212254]]
* 21:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 [[phab:T212254|T212254]] (duration: 00m 53s)
* 21:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 [[phab:T212254|T212254]] (duration: 00m 53s)
* 21:38 marostegui: Drop valid_tag table from db1123 (s3) - [[phab:T212254|T212254]]
* 21:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 [[phab:T212254|T212254]] (duration: 00m 53s)
* 21:31 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.12
* 21:03 dduvall@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache (duration: 39m 22s)
* 20:42 ejegg: updated payments-wiki from {{Gerrit|b8acb95a2a}} to {{Gerrit|c455bbc6bb}}
* 20:24 dduvall@deploy1001: Started scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache
* 20:24 gehel: starting data copy from wdqs1004 to wdqs1007 (both will be depooled) - [[phab:T213217|T213217]]
* 20:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Don't enable entities, we're not Wikidata.org (duration: 01m 44s)
* 20:11 XioNoX: change BGP_fundraising_aggregates term nat from static to aggregate on pfw3-eqiad - [[phab:T211028|T211028]]
* 19:51 ejegg: updated fundraising CiviCRM from {{Gerrit|b8e3a71845}} to {{Gerrit|5580f0b11c}}
* 19:48 krinkle@deploy1001: Finished deploy [performance/navtiming@68fd54d]: (no justification provided) (duration: 00m 05s)
* 19:48 krinkle@deploy1001: Started deploy [performance/navtiming@68fd54d]: (no justification provided)
* 19:48 dduvall@deploy1001: Pruned MediaWiki: 1.33.0-wmf.12 (duration: 06m 26s)
* 19:11 arlolra: Updated Parsoid to {{Gerrit|2c5dc7b}} ([[phab:T197616|T197616]], [[phab:T205491|T205491]], [[phab:T209772|T209772]], [[phab:T199926|T199926]], [[phab:T209194|T209194]], [[phab:T204622|T204622]])
* 19:06 marostegui: Drop valid_tag table from s1 - [[phab:T212254|T212254]]
* 19:00 arlolra@deploy1001: Finished deploy [parsoid/deploy@4b82683]: Updating Parsoid to {{Gerrit|2c5dc7b}} (duration: 10m 40s)
* 18:54 XioNoX: make pfw3-codfw source NAT similar to pfw3-eqiad - [[phab:T211028|T211028]]
* 18:54 ejegg: updated SmashPig standalone install from {{Gerrit|fb3268897b}} to {{Gerrit|25713ca232}}
* 18:50 marostegui: Drop valid_tag table from s4 - [[phab:T212254|T212254]]
* 18:50 XioNoX: add NAT workaround to pfw3-eqiad - [[phab:T211028|T211028]]
* 18:49 arlolra@deploy1001: Started deploy [parsoid/deploy@4b82683]: Updating Parsoid to {{Gerrit|2c5dc7b}}
* 18:38 XioNoX: temporarily permit ssh from frpm1001 to pfw3-eqiad on pfw3-eqiad
* 18:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 18:27 jynus: restarting s5 replication on labsdb1009/10/11
* 17:41 moritzm: installing libseccomp updates from stretch point release
* 17:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 (duration: 02m 29s)
* 17:38 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2
* 17:37 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - [[phab:T210752|T210752]] [[phab:T197616|T197616]] (duration: 96m 50s)
* 17:33 _joe_: applying the new apache configuration to jobrunners in eqiad
* 17:24 elukey: roll restart of aqs on aqs100* to pick up new Druid settings
* 17:20 _joe_: depooling mw1299 for testing of the apache change
* 17:16 SMalyshev: restarted Blazegraph wdqs1006 due to unresponsiveness (caused by load?)
* 16:56 urandom: forcing removal of restbase1016-a (host down way too long to salvage) -- [[phab:T212418|T212418]]
* 16:56 jynus: changing db1124:s5 replication to db2066
* 16:55 marostegui: Deploy schema change on db1105:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 16:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 44s)
* 16:54 jynus: stopping s5 replication on labsdb1009/10/11 to prevent undoable mistakes
* 16:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 - [[phab:T212833|T212833]] (duration: 02m 51s)
* 16:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 16:12 XioNoX: add BGP sessions to AS64050 in AMS-IX
* 16:04 marostegui: Drop valid_tag table from s7 - [[phab:T212254|T212254]]
* 16:00 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - [[phab:T210752|T210752]] [[phab:T197616|T197616]]
* 15:59 marostegui: Deploy schema change on db1089 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 15:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 15:45 marostegui: Drop valid_tag table from s2 - [[phab:T212254|T212254]]
* 15:32 marostegui: Stop MySQL on es2019 for upgrade - [[phab:T212833|T212833]]
* 15:23 godog: briefly stop carbon daemons on graphite1004 to move /srv/whisper -> /srv/carbon/whisper
* 15:17 marostegui: Increase connections from 10 to 50 for recommendationapiservice on m2 - [[phab:T212154|T212154]]
* 15:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2019 - [[phab:T212833|T212833]] (duration: 00m 44s)
* 15:04 hashar: Restarted CI Jenkins
* 13:02 zeljkof: EU SWAT finished
* 12:59 jynus: transfering db1102:s5 mariadb datadir to db1082
* 12:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481107{{!}}Give all users (including IPs) the pagequality right in plwikisource (T212478)]] (duration: 00m 45s)
* 12:45 akosiaris@deploy1001: scap-helm zotero finished
* 12:45 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
* 12:45 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 12:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481662{{!}}Allow ptwikis bureaucrats to grant/revoke rollbacker user group (T212735)]] (duration: 00m 45s)
* 12:39 akosiaris@deploy1001: scap-helm zotero upgrade production2 -f zoterov2-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481663{{!}}Use localized wgMetaNamespace and wgMetaNamespaceTalk in satwiki (T211294)]] (duration: 00m 45s)
* 12:23 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:481240{{!}}New throttle rule for students writing Wikipedia program (T212226)]] (duration: 00m 44s)
* 12:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:482586{{!}}New throttle rule for University of Southern California editathon (T212917)]] (duration: 00m 45s)
* 12:07 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: [[phab:T212768|T212768]] [cirrus] re-enable HHVM connection pooling (duration: 00m 45s)
* 12:01 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource (duration: 12m 38s)
* 11:49 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource
* 11:46 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Increase time out on the MW side to 60s - [[phab:T204183|T204183]] (duration: 00m 51s)
* 11:36 akosiaris@deploy1001: scap-helm zotero finished
* 11:36 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 11:36 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 11:35 akosiaris@deploy1001: scap-helm zotero finished
* 11:35 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 11:35 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 11:33 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron strugling, restart - [[phab:T213154|T213154]]
* 11:29 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw
* 11:24 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=zotero,name=codfw
* 11:07 jynus: stoping and restarting db1102 (s5, s4) for upgrade
* 11:04 moritzm: rebooting mw1261
* 10:48 moritzm: installing libseccomp updates from stretch point release
* 10:34 dcausse: elastic@eqiad setting crosscluster conf on production search cluster ([[phab:T213150|T213150]])
* 10:25 banyek: executing schema change on db1062 - [[phab:T85757|T85757]]
* 09:39 foks: reset user email for Zergiorubio
* 09:26 akosiaris@deploy1001: scap-helm zotero finished
* 09:26 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
* 09:26 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 09:22 jynus: stop replication on db1124:s5 [[phab:T213108|T213108]]
* 09:21 akosiaris@deploy1001: scap-helm zotero finished
* 09:21 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 09:21 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # [[phab:T101131|T101131]]
* 09:12 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - [[phab:T192691|T192691]] (duration: 00m 59s)
* 09:12 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - [[phab:T192691|T192691]]
* 05:39 SMalyshev: restarted some Blazegraph servers as precaution against corruption issues
* 04:26 onimisionipe: depooling wdqs1008 - [[phab:T213134|T213134]]
* 03:23 kartik@deploy1001: Finished deploy [cxserver/deploy@b669f95]: Update cxserver to {{Gerrit|d6b1d6f}} (duration: 05m 00s)
* 03:18 kartik@deploy1001: Started deploy [cxserver/deploy@b669f95]: Update cxserver to {{Gerrit|d6b1d6f}}
* 00:22 gehel: restarting tilerator on all maps servers
* 00:06 gehel: depooling wdqs1007 (something looks like DB corruption)
 
== 2019-01-07 ==
* 23:56 eileen: update civicrm revision changed from {{Gerrit|bcb4b7a7d1}} to {{Gerrit|b8e3a71845}}, config revision is {{Gerrit|260be32d0a}}
* 22:08 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Re-enable uploading of files, accidentally prevented (duration: 00m 44s)
* 21:19 XioNoX: push NAT changes to pfw3-eqiad - [[phab:T211028|T211028]]
* 21:16 awight@deploy1001: Finished deploy [ores/deploy@9253beb]: [[phab:T212530|T212530]]: new ORES models; revscoring 2.3.0 (duration: 15m 28s)
* 21:13 mforns@deploy1001: Finished deploy [analytics/refinery@faac592]: deploying analytics/refinery to account with refinery-source v0.0.83 (duration: 06m 52s)
* 21:06 mforns@deploy1001: Started deploy [analytics/refinery@faac592]: deploying analytics/refinery to account with refinery-source v0.0.83
* 21:00 awight@deploy1001: Started deploy [ores/deploy@9253beb]: [[phab:T212530|T212530]]: new ORES models; revscoring 2.3.0
* 20:19 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Final go-switch for WBMI {{Gerrit|Ie52b8af006ba}} (duration: 00m 45s)
* 19:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove redundant namespace talk definitions ([[phab:T206952|T206952]]) (duration: 00m 44s)
* 19:46 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgMetaNamespace for bewikibooks ([[phab:T212665|T212665]]) (duration: 00m 45s)
* 19:43 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseRepo and WikibaseMediaInfo on testcommonswiki (duration: 00m 44s)
* 19:42 XioNoX: push firewall change to pfw3-codfw/eqiad - [[phab:T211712|T211712]]
* 19:40 catrope@deploy1001: Synchronized wmf-config/Wikibase.php: Set empty clientDbList for testcommonswiki (duration: 00m 44s)
* 19:38 catrope@deploy1001: Synchronized dblists/wikidata.dblist: Enable Wikidata on testcommonswiki (duration: 00m 44s)
* 19:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add importupload to sysops on testcommons (duration: 00m 45s)
* 19:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on viwikisource ([[phab:T212929|T212929]]) (duration: 00m 45s)
* 19:13 catrope@deploy1001: Synchronized dblists/flow.dblist: Enable Flow on viwikisource ([[phab:T212929|T212929]]) (duration: 00m 45s)
* 19:11 RoanKattouw: Ran emptyUserGroup.php for autoreview, reviewer and editor groups on srwikinews ([[phab:T212058|T212058]])
* 18:51 XioNoX: re-deactivate bgp sessions to Zayo on cr1-eqiad - [[phab:T212791|T212791]]
* 18:20 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@d8f911c]: new GUI, Updater & Blazegraph build (duration: 10m 13s)
* 18:18 XioNoX: activate bgp sessions to Zayo on cr1-eqiad - [[phab:T212791|T212791]]
* 18:10 jynus: manually creating tables on es1015, es1017 with replication for testcommonswiki
* 18:10 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d8f911c]: new GUI, Updater & Blazegraph build
* 18:07 onimisionipe@deploy1001: deploy aborted: (no justification provided) (duration: 00m 04s)
* 18:06 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d8f911c]: (no justification provided)
* 18:05 XioNoX: deactivate bgp sessions to Zayo on cr1-eqiad [[phab:T212791|T212791]]
* 17:35 akosiaris: restart pdfrender on scb1004
* 17:35 akosiaris: restart pdfrender
* 17:23 kartik@deploy1001: Finished deploy [cxserver/deploy@594420b]: Update cxserver to {{Gerrit|7632c43}} (duration: 04m 06s)
* 17:19 kartik@deploy1001: Started deploy [cxserver/deploy@594420b]: Update cxserver to {{Gerrit|7632c43}}
* 16:24 jynus: shutting down mariadb again and rebooting db1107
* 16:15 jynus: starting mariadb on db1107
* 16:12 onimisionipe: starting inplace reindexing for enwiki - [[phab:T212224|T212224]]
* 16:07 volans: powercycle db1107
* 16:03 elukey: stop eventlogging mysql consumers on eventlog1002 and eventlogging replication on db1108 due to issues with db1107
* 16:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 00m 45s)
* 15:46 cmjohnson1: replacing bad fuse on the PDU rack A2 eqiad
* 14:19 moritzm: added jbond to WMF-LDAP group in Phabricator ([[phab:T213079|T213079]])
* 13:56 ariel@deploy1001: Finished deploy [dumps/dumps@acd9bca]: logging and quiet mode for adds-changes and other dumps (duration: 00m 05s)
* 13:56 ariel@deploy1001: Started deploy [dumps/dumps@acd9bca]: logging and quiet mode for adds-changes and other dumps
* 13:02 zeljkof: EU SWAT finished
* 13:01 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:480829{{!}}cirrus: increase number of shards (T212224)]] (duration: 00m 44s)
* 12:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:482583{{!}}Restrict moving categories for users at srwiki (T213050)]] (duration: 00m 44s)
* 12:40 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:482628{{!}}Cleanup old throttle rules]] (duration: 00m 44s)
* 12:34 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:482263{{!}}To lift a cap on account creation from IP for mrwiki community (T212921)]] (duration: 00m 43s)
* 12:30 Zoranzoki21: tools.zoranzoki21wiki Archived https://www.mediawiki.org/w/index.php?title=Extension:Woopra (https://www.wikidata.org/wiki/Q21679347) - [[phab:T212994|T212994]]
* 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476368{{!}}Enable reader trust survey (T209882)]] (duration: 00m 45s)
* 12:21 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481512{{!}}Enable Quiz extension on ru.wikibooks (T212622)]] (duration: 00m 45s)
* 12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481513{{!}}Add suppressredirect user right to editor user group at pl.wikisource (T212655)]] (duration: 00m 44s)
* 12:11 gtirloni: disabled notifications for cloudvirt0124 ([[phab:T212360|T212360]])
* 12:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481538{{!}} Enable extendedmover user group at en.wiktionary (T212662)]] (duration: 00m 46s)
* 12:07 kartik@deploy1001: Finished deploy [cxserver/deploy@2d54a64]: Deploy Google Translation ([[phab:T90208|T90208]]) (duration: 05m 07s)
* 12:02 kartik@deploy1001: Started deploy [cxserver/deploy@2d54a64]: Deploy Google Translation ([[phab:T90208|T90208]])
* 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1079 after schema change - [[phab:T85757|T85757]] (duration: 00m 44s)
* 10:31 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Move group1 to new logging infrastructure - [[phab:T211124|T211124]] (duration: 00m 45s)
* 10:30 banyek: repooling db1079 after schema change - [[phab:T85757|T85757]]
* 10:27 banyek: restarting replication on db1079 - [[phab:T85757|T85757]]
* 09:55 banyek: executing schema change on db1079 with replication enabled - [[phab:T85757|T85757]]
* 09:53 banyek: stopping replication on db1079 - [[phab:T85757|T85757]]
* 09:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1079 for schema change - [[phab:T85757|T85757]] (duration: 01m 02s)
* 09:36 banyek: depooling db1079 for schema change - [[phab:T85757|T85757]]
* 08:30 moritzm: rolling restart of swift backend servers to pick up OpenSSL security update
* 07:24 elukey: restart pdfrender on scb1002
 
== 2019-01-06 ==
* 14:50 ariel@deploy1001: Finished deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag (duration: 00m 06s)
* 14:50 ariel@deploy1001: Started deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag
 
== 2019-01-05 ==
* 20:23 elukey: manually clean up of big logs under /var/log/.. on analytics-tool1002 due to root partition almost filled up
 
== 2019-01-04 ==
* 23:07 mutante: scandium apt-get remove nodejs nodes-legacy ; puppet agent -tv  - after merging gerrit:482150 this fixed "you have held broken packages" issue, now we are at a puppet dependecy cycle with apt::pin [[phab:T201366|T201366]]
* 15:42 bawolff@deploy1001: Synchronized private/PrivateSettings.php: [[phab:T212667|T212667]] - More aggressive anti-spam measures for account creation on kowiki (duration: 00m 48s)
* 14:08 moritzm: rebooting etcd1001-1003 to pick up SSBD-enabled qemu
* 13:52 moritzm: rebooting etcd1004-1006 to pick up SSBD-enabled qemu
* 13:33 moritzm: rebooting kubernetes staging etcd hosts to pick up SSBD-enabled qemu
* 13:11 moritzm: rebooting kubernetes staging master to pick up SSBD-enabled qemu
* 12:57 moritzm: rebooting kubernetes staging workers for kernel security update
* 11:58 moritzm: installing libsndfile security updates
* 11:33 moritzm: installing jasper security updates
* 11:31 moritzm: installing libdatetime-timezone-perl updates for recent tz changes
* 10:47 arturo: [[phab:T212898|T212898]] reimaging cloudvirt1024 as stretch
* 10:46 moritzm: rolling restart of swift proxies to pick up OpenSSL update
* 09:57 jijiki: restarting thumbor services to pick up 481141
* 09:50 onimisionipe: restarting nginx on all wdqs hosts
* 09:40 banyek: executing schema change on dbstore1002 - [[phab:T85757|T85757]]
* 09:13 moritzm: restarting nginx on puppetdb hosts to pick up new OpenSSL
* 09:03 banyek: executing schema change on db1116 - [[phab:T85757|T85757]]
* 08:44 moritzm: restarting nginx on francium to pick up new OpenSSL
* 08:16 elukey: restart eventlogging daemons on eventlog1002 to pick up openssl updates
* 07:56 moritzm: installing OpenSSL security updates
* 00:07 mutante: an-coord1001 - apt-get clean to free disk space, reacting to Icinga alert for running out of disk
 
== 2019-01-03 ==
* 23:08 volans: restarted pdfrender on scb1004
* 22:29 volans: restarted all slaves on dbstore1002 (relayed from banyek)
* 22:14 banyek: stopping all slaves on dbstore1002 (NOT labsdb)
* 22:14 banyek: stopping all slaves on labsdb1002
* 20:50 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: Fix error for testcommons (duration: 00m 44s)
* 20:46 reedy@deploy1001: Synchronized dblists/group0.dblist: Add testcommonswiki to group0 (duration: 00m 43s)
* 20:43 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 05s)
* 20:24 reedy@deploy1001: Synchronized wmf-config/db-codfw.php: [[phab:T197616|T197616]] (duration: 00m 44s)
* 20:23 reedy@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T197616|T197616]] (duration: 00m 44s)
* 20:13 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T197616|T197616]] (duration: 00m 44s)
* 20:12 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: [[phab:T197616|T197616]] (duration: 00m 44s)
* 20:11 reedy@deploy1001: rebuilt and synchronized wikiversions files: [[phab:T197616|T197616]]
* 20:09 reedy@deploy1001: Synchronized dblists/: [[phab:T197616|T197616]] (duration: 00m 45s)
* 18:51 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@1182b3b]: Update mobileapps to {{Gerrit|f6ad0e5}}: Set timeout for backend /page/html requests, part 2 (duration: 05m 27s)
* 18:46 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@1182b3b]: Update mobileapps to {{Gerrit|f6ad0e5}}: Set timeout for backend /page/html requests, part 2
* 18:37 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@c470ed2]: Update mobileapps to {{Gerrit|f6ad0e5}}: Set timeout for backend /page/html requests (duration: 04m 11s)
* 18:33 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@c470ed2]: Update mobileapps to {{Gerrit|f6ad0e5}}: Set timeout for backend /page/html requests
* 18:21 volans: restart pdfrender on scb1003
* 17:58 ariel@deploy1001: Finished deploy [dumps/dumps@10dc8ad]: return properly if commands failed (duration: 00m 08s)
* 17:58 ariel@deploy1001: Started deploy [dumps/dumps@10dc8ad]: return properly if commands failed
* 16:32 XioNoX: remove old 10.64.22.0/24 IPs from cloud-instance-transport1-b-eqiad - [[phab:T207663|T207663]]
* 16:22 moritzm: rebooting kubernetes workers in eqiad for kernel security update
* 16:02 arturo: reimaging cloudvirt1013 cloudvirt1026-1028 to stretch
* 15:48 moritzm: restart parsoid on wtp1025 to pick up OpenSSL update for nodejs
* 15:43 jijiki: Enabled puppet on mw servers after merging 481796 - [[phab:T197616|T197616]]
* 15:31 jijiki: Disabling puppet on mw servers to test 481796 - [[phab:T197616|T197616]]
* 15:14 ejegg: updated Fundraising CiviCRM from {{Gerrit|b33dcd3c94}} to {{Gerrit|bcb4b7a7d1}}
* 14:37 moritzm: rebooting kubernetes workers in codfw for kernel security update
* 14:37 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1101:3317 after schema change - [[phab:T85757|T85757]] (duration: 00m 44s)
* 14:32 banyek: repooling db1101:3317 after schema change - [[phab:T85757|T85757]]
* 14:21 moritzm: rebooting kubernetes masters in eqiad to pick up SSBD-enabled qemu
* 14:14 moritzm: rebooting kubernetes mastes in codfw to pick up SSBD-enabled qemu
* 14:05 arturo: [[phab:T209616|T209616]] reimage cloudvirt1029 as debian stretch
* 13:43 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1101:3317 for schema change - [[phab:T85757|T85757]] (duration: 00m 44s)
* 13:41 banyek: depooling db1101:3317 for schema change - [[phab:T85757|T85757]]
* 13:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1098:3317 after schema change - [[phab:T85757|T85757]] (duration: 00m 44s)
* 13:34 banyek: repooling db1098:3317 after schema change - [[phab:T85757|T85757]]
* 13:24 kartik@deploy1001: Finished deploy [cxserver/deploy@3b2ede7]: Update cxserver to {{Gerrit|2369a18}} (duration: 04m 30s)
* 13:20 kartik@deploy1001: Started deploy [cxserver/deploy@3b2ede7]: Update cxserver to {{Gerrit|2369a18}}
* 12:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1098:3317 for schema change - [[phab:T85757|T85757]] (duration: 00m 45s)
* 12:55 banyek: depooling db1098:3317 for schema change - [[phab:T85757|T85757]]
* 12:54 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1094 after schema change - [[phab:T85757|T85757]] (duration: 00m 45s)
* 12:49 banyek: repooling db1094 after schema change - [[phab:T85757|T85757]]
* 12:41 arturo: [[phab:T212302|T212302]] reimaging again cloudvirt1030 to test final puppet code
* 12:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1094 for schema change - [[phab:T85757|T85757]] (duration: 00m 46s)
* 12:28 banyek: depooling db1094 for schema change - [[phab:T85757|T85757]]
* 12:27 moritzm: restarting tor on torrelay1001 to pick up OpenSSL security update
* 11:02 _joe_: manually reloading icinga to pick up changes to commands.cfg
* 10:55 moritzm: installing apache updates on puppetmasters
* 10:22 moritzm: installing ghostscript security updates on jessie
* 09:51 elukey: restart memcached on mc1023 to apply -R 200 - [[phab:T208844|T208844]]
* 09:46 moritzm: remove imagemagick remnants from ATS hosts (obsoleted by upstream packaging change which dropped the webp plugin)
* 09:39 moritzm: installing nginx updates on puppetdb*
* 09:26 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2019 - [[phab:T212833|T212833]] (duration: 01m 33s)
* 09:18 banyek: repooling es2019 - [[phab:T212833|T212833]]
* 08:46 moritzm: rolling restart of proton to pick up OpenSSL update
* 08:35 banyek: depooled es2019 as host was unsresponsive - [[phab:T212833|T212833]]
* 08:35 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2019, host is unsresponsible - [[phab:T212833|T212833]] (duration: 00m 49s)
* 08:11 moritzm: installing OpenSSL security updates
* 00:21 mutante: notebook1004 - started nagios-nrpe-server one more time
 
== 2019-01-02 ==
* 23:59 mutante: notebook1004 still keeps running out of memory from some user actions and that kills nagios-nrpe-server and that causes a bunch of Icinga alerts
* 23:39 mutante: notebook1004 - systemctl start nagios-nrpe-server
* 23:39 mutante: notebook1004 - systemctl status nagios-nrpe-server
* 20:59 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=parsoid,service=parsoid,name=wtp1028.eqiad.wmnet
* 20:59 herron: repooling wtp1028 [[phab:T212624|T212624]]
* 20:52 herron: rebooting wtp1028 — looking for POST errors [[phab:T212624|T212624]]
* 20:05 Krinkle: mwmaint1002: foreachwikiindblist s5 deleteEqualMessages.php
* 20:04 Krinkle: mwmaint1002: foreachwikiindblist s2 deleteEqualMessages.php
* 18:35 volans: restarting icinga on icinga1001 [[phab:T212669|T212669]]
* 16:50 XioNoX: create BGP sessions to AS3214 in AMS-IX
* 16:46 XioNoX: remove BGP sessions to AS42949 in AMS-IX (leaving the IX)
* 16:43 XioNoX: remove BGP sessions to AS6866 in AMS-IX (leaving the IX)
* 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1090:3317 after schema change - [[phab:T85757|T85757]] (duration: 00m 46s)
* 16:30 arturo: reimaging cloudvirt1030 with stretch, server cleanup after puppet refactoring
* 16:29 moritzm: restarting Superset to pick up openssl security update
* 16:25 moritzm: restarting Hue to pick up openssl security update
* 16:23 arturo: [[phab:T212302|T212302]] re-enable puppet in all {cloud,lab}virt* servers, all was fine
* 16:22 banyek: repooling db1090:3317 after schema change ([[phab:T85757|T85757]])
* 16:11 arturo: [[phab:T212302|T212302]] disable puppet in all {cloud,lab}virt* servers to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/481194/
* 15:39 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1090:3317 for schema change - [[phab:T85757|T85757]] (duration: 00m 44s)
* 15:34 moritzm: installing OpenSSL security updates
* 15:31 banyek: depooling db1090:3317 for schema change ([[phab:T85757|T85757]])
* 15:13 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1086 after schema change - [[phab:T85757|T85757]] (duration: 00m 44s)
* 15:07 banyek: repooling db1086 after schema change ([[phab:T85757|T85757]])
* 14:49 banyek: executing schema change on db1086 - [[phab:T85757|T85757]]
* 14:48 moritzm: installing ghostscript security update for jessie
* 14:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1086 for schema change - [[phab:T85757|T85757]] (duration: 00m 45s)
* 14:38 banyek: depooling db1086 for schema change ([[phab:T85757|T85757]])
* 14:15 ema: cp hosts: upgrade OpenSSL from 1.1.0f to 1.1.0j
* 13:39 moritzm: installing ghostscript update for stretch
* 13:33 moritzm: installing libav security updates
* 13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 44s)
* 13:17 moritzm: installing openjpeg2 security updates
* 13:17 banyek: executing schema change on db2040 (s7 codfw master) replication lag could be expected on codfw - [[phab:T85757|T85757]]
* 13:13 banyek: stopping replication on db2077 prior to executing schema change on codfw s7 master (db2040) - [[phab:T85757|T85757]]
* 13:06 marostegui: Deploy schema change on db1119 - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 13:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 13:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 12:00 moritzm: rebooting labtestpuppetmaster2001 for kernel security update
* 11:53 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe1006.eqiad.wmnet
* 11:51 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.eqiad.wmnet
* 11:50 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.codfw.wmnet
* 11:46 ema: replace TLS certificates on ms-fe eqiad hosts [[phab:T212215|T212215]]
* 11:41 moritzm: rebooting labtestweb2001 for kernel security update
* 11:24 marostegui: Deploy schema change on db1099:3311 - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 11:17 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
* 11:10 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
* 10:59 ema: replace TLS certificates on ms-fe codfw hosts [[phab:T212215|T212215]]
* 10:52 moritzm: rebooting centrallog1001 for kernel security update
* 10:48 volans: testing the new spicerack package on cumin2001, in the unlikely event you need to use spicerack cookbooks today please use cumin1001
* 10:45 godog: ms-be2018 Flashing Smart Array P840 in Slot 3 [ 3.00 -> 6.60 ]
* 10:43 moritzm: removed labvirt1013 from debmonitor, got renamed in [[phab:T212513|T212513]]
* 10:42 volans: uploaded spicerack_0.0.10-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
* 10:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2096 (duration: 00m 44s)
* 09:50 marostegui: Stop MySQL on db2096 for kernel and mysql upgrade
* 09:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2096 (duration: 00m 45s)
* 09:48 marostegui@deploy1001: sync-file aborted: Depool db2096 (duration: 00m 01s)
* 09:18 moritzm: installing c3p0 security updates
* 09:07 Zoranzoki21: Drop valid_tag from s8 by Marostegui - [[phab:T212254|T212254]]
* 09:06 godog: eqiad-prod: final weight for ms-be10[44-50].eqiad.wmnet - [[phab:T209618|T209618]]
* 08:56 moritzm: installing libarchive security updates
* 07:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - [[phab:T212692|T212692]] (duration: 00m 46s)
* 07:30 marostegui: Fix login.logging table on db1078 - [[phab:T212692|T212692]]
* 07:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - [[phab:T212692|T212692]] (duration: 00m 47s)
* 07:01 marostegui: Deploy schema change on s1 codfw master (lag will be generated on s1 codfw) - [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 06:54 marostegui: Drop empty valid_tag table from labswiki labtestwiki - [[phab:T212254|T212254]]
* 06:49 marostegui: Drop empty valid_tag table from s5 - [[phab:T212254|T212254]]
* 06:25 marostegui: Drop valid_tag from s6 - [[phab:T212254|T212254]]
* 06:15 marostegui: Fix last chunks on db1124:338 - [[phab:T212574|T212574]]


__NOTOC__
__NOTOC__

Revision as of 01:10, 3 February 2019

2019-02-03

  • 01:10 elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred."

2019-02-02

  • 20:42 chaomodus: restarted pdfrender on scb1003
  • 20:41 chaomodus: restarted pdfrender on scb1004
  • 20:06 chaomodus: parsoid was failed on scandium and alerting, the service parsoid-vd was restarted and appears to have come back
  • 05:44 jforrester@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/VisualEditor/lib/ve/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js: b/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js T214963 Hot-deploy VE fix to stop hitting user pref writes without debounce (duration: 01m 02s)

2019-02-01

  • 23:16 vgutierrez: restart pdfrender on scb1004
  • 21:57 ejegg: updated payments-wiki-staging from 7767c7027e to 52a271e681
  • 21:25 ejegg: updated payments-wiki-staging to fundraising/REL1_31 branch
  • 07:13 bawolff_: reset 2FA on wikitech for User:Cicalese


Archives

See Server admin log/Archives.