Renaming of Cloud Services products is in progress.
You may see mixed references to Tool Labs -> Toolforge, Labs -> Cloud VPS. Edits are welcome.

Server Admin Log

From Wikitech
Jump to: navigation, search

2017-12-13

  • 00:49 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)
  • 00:44 ebernhardson@tin: Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/: SWAT: T182616: turn on second mlr ab test for hewiki (duration: 01m 08s)
  • 00:40 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on cirrus MLR for most wikis with >1% of search traffic (duration: 01m 08s)
  • 00:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588 (duration: 04m 48s)
  • 00:32 brion: restarted requeueTranscodes.php on terbium for mp3 audio generation backfill (had dropped DB connection)
  • 00:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@5832a8c]: Update mobileapps to bfc3588
  • 00:05 ebernhardson@tin: Synchronized wmf-config/: SWAT: T182616: Setup MLR AB test for hewiki (duration: 01m 10s)

2017-12-12

  • 22:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b (duration: 05m 10s)
  • 22:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@ea8f05d]: Update mobileapps to 94f267b
  • 22:02 urandom: setting compaction throughput to 5 MB/s, restbase1010
  • 21:31 mutante: releases1001 - rm mediawiki core repo and let puppet try to recreate it (follow-up issue after gerrit:397891)
  • 21:28 mobrovac@tin: Finished deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417 (duration: 04m 16s)
  • 21:27 demon@tin: Synchronized php-1.31.0-wmf.12/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 07s)
  • 21:26 mholloway-shell@tin: Finished deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651 (duration: 01m 45s)
  • 21:24 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/GlobalBlocking/: (no justification provided) (duration: 01m 08s)
  • 21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@28bfda3]: Update mobileapps to d0ee651
  • 21:24 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7 (duration: 00m 30s)
  • 21:24 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
  • 21:23 mobrovac@tin: Started deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417
  • 21:21 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4 (duration: 02m 45s)
  • 21:18 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 4
  • 21:15 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3 (duration: 02m 04s)
  • 21:13 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 3
  • 21:13 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender (duration: 05m 05s)
  • 21:12 mobrovac@tin: Started restart [electron-render/deploy@94d27d7]: Electron hanging - T174916
  • 21:08 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries, take 2 after failed content rerender
  • 21:08 ppchelko@tin: Finished deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries (duration: 05m 51s)
  • 21:02 ppchelko@tin: Started deploy [restbase/deploy@dceab2e]: Bump expected parsoid version, but do not switch summaries
  • 20:55 ppchelko@tin: Finished deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries (duration: 01m 36s)
  • 20:53 ppchelko@tin: Started deploy [restbase/deploy@d3ca789]: Revert deployment for using MCS for summaries
  • 20:53 demon@tin: rebuilt and synchronized wikiversions files: group0 to wmf.12
  • 20:51 ppchelko@tin: Finished deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS (duration: 03m 30s)
  • 20:48 ppchelko@tin: Started deploy [restbase/deploy@506047c]: Update expected Parsoid version, switched summary to MCS
  • 20:44 mholloway-shell@tin: Finished deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7 (duration: 04m 48s)
  • 20:39 mholloway-shell@tin: Started deploy [mobileapps/deploy@b2d5b8e]: Update mobileapps to 172abc7
  • 20:35 mholloway-shell@tin: Finished deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d (duration: 02m 32s)
  • 20:32 mholloway-shell@tin: Started deploy [mobileapps/deploy@0a9d635]: Update mobileapps to 035608d
  • 20:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d (duration: 40m 04s)
  • 19:57 arlolra: Updated Parsoid to 741fc5d (T114072, T181226, T21910, T152540, T103714, T97093, T118520, T181229, T182338, T182170, T169006)
  • 19:51 mholloway-shell@tin: Started deploy [mobileapps/deploy@2690678]: Update mobileapps to 5b8796d
  • 19:43 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: (no justification provided) (duration: 14m 57s)
  • 19:35 anomie: Running cleanupUsersWithNoId.php on all wikis (this will take a while), see T181731
  • 19:12 arlolra: Parsoid deploy aborted and rolled back to 01c1fc3 while RESTBase fixes an issue
  • 19:06 arlolra@tin: Finished deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d (duration: 05m 33s)
  • 19:00 arlolra@tin: Started deploy [parsoid/deploy@98139cb]: Updating Parsoid to 741fc5d
  • 18:48 aaron@tin: Synchronized php-1.31.0-wmf.12/includes/Setup.php: 058c17e702eb0 (duration: 01m 09s)
  • 18:21 moritzm: uploaded prometheus-ircd-exporter to apt.wikimedia.org
  • 17:52 demon@tin: Finished scap: bootstrap wmf.12 (duration: 29m 35s)
  • 17:22 demon@tin: Started scap: bootstrap wmf.12
  • 17:15 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 (duration: 08m 45s)
  • 17:12 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 03s)
  • 17:12 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 17:11 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 36s)
  • 17:11 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 17:03 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 16:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 and db1081 original weight (duration: 00m 56s)
  • 16:40 jynus: restart and upgrade db1059 (phabricator passive db)
  • 16:35 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001 (duration: 00m 19s)
  • 16:35 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2001
  • 16:34 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001 (duration: 00m 20s)
  • 16:34 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2001
  • 16:30 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002 (duration: 00m 20s)
  • 16:30 gehel@tin: Started deploy [tilerator/deploy@29d633e]: new tilerator packaging on maps-test2002
  • 16:28 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002 (duration: 00m 18s)
  • 16:28 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2002
  • 16:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 53s)
  • 16:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 55s)
  • 16:22 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 16:21 akosiaris: failover boron to ganeti1008
  • 16:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 (duration: 00m 56s)
  • 15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 04s)
  • 15:59 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 15:59 akosiaris@tin: Finished deploy [ores/deploy@b4f2b02]: T181661 (duration: 00m 09s)
  • 15:58 akosiaris@tin: Started deploy [ores/deploy@b4f2b02]: T181661
  • 15:46 jynus: stop, upgrade and reboot db2072
  • 15:34 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2072 (duration: 00m 56s)
  • 15:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 - T174569 (duration: 01m 01s)
  • 15:31 marostegui: Deploy schema change on s4 db1097:3314 - T174569
  • 15:25 gehel: powercycling maps-test2003
  • 15:24 elukey: rename notebook1002 -> kafka1023 - step 3, replace notebook1002 with kafka1023 in the puppet config
  • 15:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight (duration: 01m 18s)
  • 15:06 marostegui: Upgrade MySQl on db1084
  • 15:02 elukey: clear recdns records related to notebook1002/kafka1023 (rec_control wipe-cache kafka1023.eqiad.wmnet kafka1023.mgmt.eqiad.wmnet notebook1002.eqiad.wmnet 14.5.64.10.in-addr.arpa 104.3.65.10.in-addr.arpa) - T181518
  • 14:58 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
  • 14:46 elukey: start rename notebook1002 -> kafka1023 - step 2, dns config (host already shutdown) - T181518
  • 14:39 hashar: Rebuild operations-puppet-tests-docker image based on c76d892 | T178620 and /cache being owned by root
  • 14:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 56s)
  • 14:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1086 (duration: 00m 56s)
  • 14:20 zeljkof: EU SWAT finished
  • 14:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS aliases for zh_wikiquote (T181374) (duration: 00m 56s)
  • 14:09 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Lift account registration on en.wiki (T182665) (duration: 00m 56s)
  • 14:02 herron: upgrading codfw puppet agents
  • 13:59 jynus: stop, upgrade and reboot db2071
  • 13:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 56s)
  • 13:56 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 57s)
  • 13:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1086 (duration: 00m 57s)
  • 12:44 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 12:10 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 12:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
  • 11:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low weight - T163190 (duration: 00m 55s)
  • 11:27 jynus: stop, upgrade and reboot db1092
  • 11:26 marostegui: Upgrade MySQL and kernel on db1086
  • 11:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 56s)
  • 11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2066 (duration: 00m 57s)
  • 11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174569 (duration: 00m 57s)
  • 11:11 marostegui: Deploy schema change on db1084 (s4) - T174569
  • 11:05 marostegui: Deploy schema change on db1056 (s4) already depooled - T174569
  • 11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174569 (duration: 00m 56s)
  • 10:46 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003 (duration: 00m 10s)
  • 10:46 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2003
  • 10:26 jynus: stop, upgrade and reboot db2066
  • 10:24 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004 (duration: 00m 18s)
  • 10:24 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: new kartotherian packaging on maps-test2004
  • 10:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2066 (duration: 00m 56s)
  • 10:14 moritzm: reimaging mw1260 (video scaler) to stretch
  • 10:04 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2059 (duration: 00m 56s)
  • 09:51 moritzm: updated stretch installer netboot image after stretch 9.3 point release
  • 09:51 moritzm: updated stretch installer netboot image after jessie 8.10 point release
  • 09:44 marostegui: Stop db1039 and db1086 in sync - T163190
  • 09:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T163190 (duration: 00m 56s)
  • 09:38 gehel: reduce replication factor for cassandra on maps-test cluster and reset cassandra on maps-test2001 to work around limited disk space - T182583
  • 09:34 marostegui: Stop replication in sync on db1034 and db1039 for data consistency check - T163190
  • 09:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after InnoDB there - T178359 (duration: 00m 56s)
  • 09:00 jynus: stop, upgrade and reboot db2059
  • 08:57 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
  • 08:47 moritzm: updated jessie installer netboot image after jessie 8.10 point release
  • 07:15 marostegui: Deploy schema change on db1081 - https://phabricator.wikimedia.org/T174569
  • 07:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174569 (duration: 00m 56s)
  • 06:56 marostegui: stop MySQL on db1055 - T182653
  • 06:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 T182653 (duration: 00m 56s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 06m 21s)
  • 00:53 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Remove manual firejailing of Score binaries (T181535) (duration: 00m 56s)
  • 00:47 legoktm@tin: Synchronized wmf-config/CommonSettings.php: Have ExtensionDistributor treat REL1_30 as stable (duration: 00m 56s)
  • 00:39 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable ORES on fawiki (T182354) (duration: 00m 56s)
  • 00:25 ejegg: updated CiviCRM from 2db9c76 to 85a8526
  • 00:23 catrope@tin: Synchronized wmf-config/CommonSettings.php: Fix default for rcOresDamagingPref (duration: 00m 56s)
  • 00:22 ejegg: disabled nightly Ingenico WX audit parsing job

2017-12-11

  • 23:37 smalyshev@tin: Finished deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update (duration: 06m 10s)
  • 23:31 smalyshev@tin: Started deploy [wdqs/wdqs@f6b110f]: updater fix and GUI update
  • 23:28 mholloway-shell@tin: Finished deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17 (duration: 06m 16s)
  • 23:22 mholloway-shell@tin: Started deploy [mobileapps/deploy@07293bc]: Update mobileapps to e290b17
  • 21:11 mholloway-shell@tin: Finished deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333 (duration: 07m 56s)
  • 21:04 mholloway-shell@tin: Started deploy [mobileapps/deploy@6347d62]: Update mobileapps to 61ca333
  • 19:47 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T181493: Enable Page Previews EventLogging instrumentation (duration: 00m 56s)
  • 19:47 mobrovac@tin: Finished deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107 (duration: 06m 19s)
  • 19:46 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID (duration: 00m 29s)
  • 19:46 ppchelko@tin: Started deploy [cpjobqueue/deploy@b1beaf1]: Revert dedupe based on sha1 as well as on event ID
  • {{safesubst:SAL entry|1=19:43 urandom: lower compaction throughput to 2 MB/s, restbase1010-{a,b,c} - T178177}}
  • 19:40 mobrovac@tin: Started deploy [restbase/deploy@be7d72f]: Expose the Reading Lists end points, take #3 - T181107
  • 19:34 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107 (duration: 05m 33s)
  • 19:28 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points, take #2 - T181107
  • 19:28 mobrovac@tin: Finished deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107 (duration: 01m 26s)
  • 19:26 mobrovac@tin: Started deploy [restbase/deploy@bce2885]: Expose the Reading Lists end points - T181107
  • 19:24 ebernhardson@tin: Synchronized wmf-config/throttle.php: SWAT: T182613 Update throttle rule for McGill University Library (duration: 00m 56s)
  • 19:21 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T131132: Switch submit button from save to publish on enwiki (duration: 02m 43s)
  • 19:19 awight@tin: Finished deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661 (duration: 01m 12s)
  • 19:17 awight@tin: Started deploy [ores/deploy@1c0ede0]: (non-production) Testing parallel ORES deployment, T181661
  • 19:16 ebernhardson@tin: Synchronized php-1.31.0-wmf.11/extensions/CirrusSearch/includes/Search/RescoreBuilders.php: SWAT: Add query string for running Cirrus MLR pre-deploy checks (duration: 00m 57s)
  • 18:56 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch cebwiki, ruwiki and small projects to Kafka for htmlCacheUpdate - T182023 (duration: 00m 57s)
  • 18:55 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023 (duration: 00m 34s)
  • 18:54 ppchelko@tin: Started deploy [cpjobqueue/deploy@e1075af]: Enable htmlCacheUpdate for ceb and ru wiki and small projects T182023
  • 18:03 gehel@tin: Finished deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates (duration: 01m 14s)
  • 18:02 gehel@tin: Started deploy [wdqs/wdqs@353b3cb]: wdqs: GUI and updater updates
  • 16:42 hashar: Restarting Jenkins
  • 15:36 marostegui: Deploy schema change on s2 master (db1054) - T174569
  • 14:51 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the wgTranslateNumerals at hiwikiversity T182584 (duration: 00m 56s)
  • 14:46 niharika29@tin: Synchronized php-1.31.0-wmf.11/includes/specials/SpecialUndelete.php: Revert replacing textarea in Special:Undelete with OOUI T182398 (duration: 00m 57s)
  • 14:34 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Bureaucrats to grant and remove translationadmin rights; sysops to add and remove the same from themselves - T182492 (duration: 00m 56s)
  • 14:32 niharika29@tin: Synchronized wmf-config/interwiki.php: Update the Interwiki map - T182506 (duration: 00m 56s)
  • 14:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Create NS_PROJECT and NS_PROJECT_TALK alias for kowikisource (T182487) (duration: 00m 56s)
  • 13:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174569 (duration: 04m 42s)
  • 12:39 _joe_: trying to get a full core dump from hhvm on mw1200
  • 11:12 _joe_: depooling mw1200 for investigation instead
  • 11:11 _joe_: restarting hhvm on mw1200, stuck in a kernel task
  • 11:08 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 11:07 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 44s)
  • 10:49 ema: cp4021: restart varnish-be due to mbox lag
  • 10:04 godog: upgrade grafana to 4.6.2 on labmon1001 - T182294
  • 10:00 jynus: stopping dbstore2001:s5 and dbstore1002 (s5) mysql replication in sync
  • 09:28 akosiaris: upload scap_3.7.4-1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:16 gehel: cleaning old cassandra dumps on maps-test2001 servers
  • 09:15 gehel: cleaning up old postgres logs on maps-test2001
  • 09:05 elukey: set notebook1002 as role::spare as prep step to reimage it to kafka1023
  • 09:03 jynus: dropping multiple leftover files from db1102
  • 08:52 marostegui: Stop replication in sync on db1034 and db1039 - T163190
  • 08:12 elukey: powercycle ganeti1008 - all vms stuck, console com2 showed a ton of printks without a clear indicator of the root cause
  • 07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T182556 (duration: 00m 45s)
  • 07:44 _joe_: restarting hhvm on mw1189,mw1229,mw1235,mw1282,mw1285,mw1315,mw1316, all stuck with a kernel hang
  • 06:59 _joe_: restarted hhvm, nginx on mw1280, hanging kernel operations
  • 06:45 marostegui: Deploy schema change on s2 db1060 with replication enabled, this will generate some lag on s2 on labs - T174569
  • 06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174569 (duration: 00m 44s)
  • 06:22 marostegui: Compress s6 on db1096 - T178359
  • 06:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 to compress InnoDB there - T178359 (duration: 00m 45s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.11) (duration: 09m 21s)

2017-12-10

  • 20:33 elukey: execute restart-hhvm on mw1312 - hhvm stuck multiple times queueing requests
  • 20:01 elukey: ran kafka preferred-replica-election for the kafka analytics cluster (1012->1022) to re-add kafka1012 to the kafka brokers acting as partition leaders (will spread the load in a better way)

2017-12-09

  • 17:00 apergos: restarted hhvm on mw1276, the same old hang with the same old symptoms
  • 16:10 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!) (duration: 03m 01s)
  • 16:07 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (take 4\!)
  • 16:02 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 05m 58s)
  • 15:56 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
  • 15:55 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 17s)
  • 15:55 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
  • 15:53 awight@tin: Finished deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity (duration: 00m 31s)
  • 15:53 awight@tin: Started deploy [ores/deploy@1c0ede0]: Reducing ORES Celery log verbosity
  • 15:53 apergos: did same on scb1002,3,4
  • 15:48 awight: Making an emergency deployment to ORES logging config to reduce verbosity.
  • 15:45 apergos: on scb1001 moved daemon.log out of the way, did "service rsyslog rotate", saved the last 5000 entries for use by ores team, removed the log
  • 11:44 apergos: that server list: mw1278, 1277, 1226, 1234, 1230
  • 11:42 apergos: restarted hhvm on api servers after lockup
  • 11:19 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Disable ORES in fawiki - T182354 (duration: 00m 45s)
  • 00:11 Jamesofur: removed 2FA from EVinente after verification T182373

2017-12-08

  • 23:23 hashar: force ran puppet on contint2001
  • 22:15 madhuvishy: Kicked off rsync of /data/xmldatadumps/public to labstore1006 & 7
  • 22:05 smalyshev@tin: Finished deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464, better fix coming soon (duration: 05m 55s)
  • 21:59 smalyshev@tin: Started deploy [wdqs/wdqs@353b3cb]: temporary fix for T182464, better fix coming soon
  • 20:22 aaron@tin: Synchronized php-1.31.0-wmf.11/includes/Setup.php: a319c3e7ab61 - disable cpPosTime injection (duration: 00m 45s)
  • 18:00 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable GlobalBlocking on fishbowl wikis (duration: 00m 45s)
  • 16:23 urandom: starting cassandra, restbase1010 - T178177
  • 16:22 urandom: disabling smart path, restbase1010, arrays 'b'...'e' - T178177
  • 16:20 urandom: disabling smart path, restbase1010, array 'a' (canary) - T178177
  • 16:15 urandom: shutting down cassandra, restbase1010 - T178177
  • 15:35 marostegui: Fix dbstore1002 s5 replication
  • 15:28 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 03s)
  • 15:28 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
  • 15:08 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 02m 08s)
  • 15:06 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
  • 15:05 gehel@tin: Finished deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003 (duration: 00m 42s)
  • 15:05 gehel@tin: Started deploy [tilerator/deploy@29d633e]: testing new tilerator packaging on maps-test2003
  • 14:39 gehel@tin: Finished deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003 (duration: 02m 34s)
  • 14:36 gehel@tin: Started deploy [tilerator/deploy@e52ea1d]: testing new tilerator packaging on maps-test2003
  • 11:45 elukey: updated prometheus-druid-exporter on druid* to 0.6
  • 11:39 elukey: upload prometheus-druid-exporter 0.6 to stretch/jessie wikimedia
  • 06:52 marostegui: Fix labsdb1004 replication broken
  • 06:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3311 - T178359 (duration: 00m 55s)
  • 04:56 bd808: scholarships: updated db schema for 2018 cycle (T181072)
  • 00:04 bblack: cp4026 - restart varnish backend, mailbox lag

2017-12-07

  • 23:59 tgr@tin: Synchronized wmf-config/InitialiseSettings.php: T181107 enable ReadingLists on all wikis (duration: 00m 46s)
  • 23:54 tgr: ran mwscript extensions/ReadingLists/maintenance/populateProjectsFromSiteMatrix.php --wiki=testwiki
  • 23:47 tgr@tin: Finished scap: T181107 deploy ReadingLists to testwiki (duration: 24m 44s)
  • 23:22 tgr@tin: Started scap: T181107 deploy ReadingLists to testwiki
  • 23:04 tgr: ran mwscript ../../../home/tgr/sql.php --wiki=mediawikiwiki --cluster extension1 --wikidb wikishared /srv/mediawiki-staging/php-1.31.0-wmf.11/extensions/ReadingLists/sql/readinglists.sql
  • 22:33 tgr@tin: Synchronized php-1.31.0-wmf.11/extensions/ReadingLists/: ReadingLists/wmf.11: catching up with master (duration: 00m 45s)
  • 22:29 tgr@tin: Synchronized php-1.31.0-wmf.10/extensions/ReadingLists/: ReadingLists/wmf.10: catching up with master (duration: 00m 46s)
  • 20:39 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.11
  • 20:35 elukey: restart hhvm on mw1235 - hhvm-dump-debug hanging out, not stacktrace available
  • 20:31 elukey: restart hhvm on mw1281 - hhvm stuck (hhvm-dump-debug timing out)
  • 20:22 joal@tin: Finished deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight) (duration: 05m 21s)
  • 20:18 herron: re-pooling eqiad puppet 4 masters via dns puppet.eqiad.wmnet puppet.wikimedia.org
  • 20:17 joal@tin: Started deploy [analytics/refinery@53bd630]: Regular analytics deploy - Long time no see, deployment :) - post-patch-2 (hopefully last for tonight)
  • 20:14 joal@tin: Finished deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch (duration: 01m 20s)
  • 20:12 joal@tin: Started deploy [analytics/refinery@3e52903]: Regular analytics deploy - Long time no see, deployment :) - post-patch
  • 19:52 joal@tin: Finished deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :) (duration: 12m 15s)
  • 19:40 joal@tin: Started deploy [analytics/refinery@bd9c6cc]: Regualr analytics deploy - Long time no see, deployment :)
  • 19:35 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start description usage tracking for commonswiki T106287 (duration: 00m 48s)
  • 19:29 thcipriani@tin: Synchronized php-1.31.0-wmf.11/includes/specialpage/ChangesListSpecialPage.php: SWAT: WLFilters: Correctly check if RCFilters should be enabled on WL T182318 (duration: 00m 48s)
  • 19:24 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Remove obsolete WikibaseQualityConstraints settings (duration: 00m 48s)
  • 19:14 ppchelko@tin: Finished deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting (duration: 01m 15s)
  • 19:13 ppchelko@tin: Started deploy [changeprop/deploy@3c4f51d]: Long awaited deploy: generic optimizations, gc metric, delay reporting
  • 19:12 thcipriani@tin: Synchronized wmf-config/throttle.php: SWAT: Rm all past throttle overrides in throttle.php (duration: 00m 48s)
  • 18:06 mholloway-shell@tin: Finished deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c (duration: 05m 36s)
  • 18:00 mholloway-shell@tin: Started deploy [mobileapps/deploy@2fa32ed]: Update mobileapps to 71f581c
  • 18:00 herron: re-pooling eqiad puppet 4 masters as puppet.ulsfo.wnet
  • 17:33 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/VisualEditor/lib/ve: Ief480487, Deskana made me do it (duration: 00m 49s)
  • 17:25 milimetric@tin: Finished deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided) (duration: 07m 28s)
  • 17:25 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1314.eqiad.wmnet
  • 17:25 herron: puppetmaster1001 upgraded to puppet 4. re-enabling puppet agents across the fleet
  • 17:18 milimetric@tin: Started deploy [analytics/aqs/deploy@4ec13b4]: (no justification provided)
  • 17:13 herron: upgrading puppetmaster1001 to puppet 4
  • 17:08 herron: temporarily disabling all puppet agents during puppetmaster1001 (puppet ca) upgrade to puppet 4
  • 16:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
  • 16:13 herron: upgrading puppetmaster1002 to puppet 4
  • 16:03 moritzm: uploaded openssl 1.0.2n for jessie-wikimedia to apt.wikimedia.org
  • 16:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 back as main traffic in s6 - T178359 (duration: 00m 48s)
  • 15:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
  • 15:42 elukey: hhvm-dump-debug for mw1314 saved to /tmp/hhvm.17991.bt.
  • 15:30 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1314.eqiad.wmnet
  • 15:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)
  • 15:19 otto@tin: Finished deploy [analytics/superset/deploy@f0f5adf]: initial deployment (duration: 00m 02s)
  • 15:18 otto@tin: Started deploy [analytics/superset/deploy@f0f5adf]: initial deployment
  • 14:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3311 with low weight - T178359 (duration: 00m 47s)
  • 14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1074 (duration: 00m 47s)
  • 14:09 zeljkof: EU SWAT finished
  • 14:08 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: alswiki: Set wgRestrictDisplayTitle = false (T182154) (duration: 00m 49s)
  • 13:39 moritzm: reimaging mw2246 to stretch
  • 12:54 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2152.codfw.wmnet
  • 12:32 addshore@tin: Synchronized wmf-config/Wikibase.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 2/2 (duration: 00m 48s)
  • 12:31 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Move Wikibase dispatchingLockManager to InitialiseSettings PT 1/2 (duration: 00m 48s)
  • 11:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
  • 11:41 moritzm: reimaging mw2152 to stretch
  • 11:35 marostegui: Compress s8 on db1099 - T178359
  • 11:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1074 (duration: 00m 48s)
  • 11:11 mobrovac@tin: Finished deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103 (duration: 05m 24s)
  • 11:05 mobrovac@tin: Started deploy [restbase/deploy@097ba7d]: Add CORS headers to erroneous responses as well - T182103
  • 10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 with low weight (duration: 00m 48s)
  • 10:50 elukey: powercycle analytics1003 - no serial console, ssh stuck in System is booting up. See pam_nologin(8)
  • 10:28 _joe_: depooling mw1283 for further investigation
  • 10:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1098:3316 db1098:3317 - T178359 (duration: 00m 51s)
  • 10:12 elukey: reboot analytics1003 for kernel+jvm updates - T179943
  • 10:01 gehel: upgrade of ELK stack on logstash100* completed - Kibana was unavailable for longer than expected - T178412
  • 09:48 hashar: CI: removed Wikidata from configuration, replaced by Wikibase. wmf/* and REL branches are going to be broken though | https://gerrit.wikimedia.org/r/395704 | T181838
  • 09:46 akosiaris: silence ganeti1006 on icinga T181121
  • 09:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 52s)
  • 09:28 marostegui: Upgrade MySQL and kernel on db1074
  • 09:22 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1009.eqiad.wmnet
  • 09:19 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1009.eqiad.wmnet
  • 09:18 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1008.eqiad.wmnet
  • 09:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1008.eqiad.wmnet
  • 09:13 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=logstash1007.eqiad.wmnet
  • 09:08 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
  • 09:05 gehel@tin: Finished deploy [logstash/plugins@b13d2fa]: (no justification provided) (duration: 00m 02s)
  • 09:05 gehel@tin: Started deploy [logstash/plugins@b13d2fa]: (no justification provided)
  • 09:00 gehel: upgrading ELK stack on logstash100* - some log messages might be lost during the upgrade - T178412
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
  • 08:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
  • 08:28 elukey: install prometheus-druid-exporter 0.5 on druid*
  • 08:26 elukey: upload prometheus-druid-exporter 0.5-1 to jessie/stretch-wikimedia
  • 07:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 47s)
  • 07:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)
  • 06:45 marostegui: Stop replication on db1099:3311 to reimport: change_tag, tag_summary, user and watchlist tables and recompress again - T178359
  • 06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174569 (duration: 00m 48s)
  • 06:40 marostegui: Deploy schema change on db1074 (s2) - T174569
  • 03:00 catrope@tin: Synchronized php-1.31.0-wmf.11/resources/src/mediawiki.rcfilters/: T182268 (duration: 00m 57s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 57s)

2017-12-06

  • 23:45 eileen: update CiviCRM from 85263f6 to 2db9c76 (improve org merges)
  • 23:12 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: (no justification provided)
  • 23:04 eileen: update process-control to b1cd515 (reenable bounce processing, enable dedupe on orgs)
  • 22:58 ppchelko@tin: Finished deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216 (duration: 00m 31s)
  • 22:57 ppchelko@tin: Started deploy [cpjobqueue/deploy@761524f]: Separate delay and totaldelay metrics T182216
  • 22:55 mutante: stat1003 - re-enabled puppet after putting role::spare on it (T175150)
  • 22:48 mutante: stat1003 - this host was kind of invisible (not in site, not in icinga) but still up, re-enabling puppet after re-adding it to site
  • 22:03 demon@tin: Synchronized php-1.31.0-wmf.11/extensions/WikidataPageBanner/includes/WikidataPageBanner.hooks.php: unbreak (duration: 00m 48s)
  • 22:00 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11, again
  • 21:51 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/lib/includes/Changes/EntityChange.php: Make EntityChange truly forward compatible with compact diffs (T182243) (duration: 00m 48s)
  • 21:43 arlolra: Updated Parsoid to 01c1fc3 (T178253, T61840, T180930, T179259)
  • 21:34 arlolra@tin: Finished deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3 (duration: 09m 45s)
  • 21:24 arlolra@tin: Started deploy [parsoid/deploy@dfcc622]: Updating Parsoid to 01c1fc3
  • 20:45 marostegui: Add 400G to labsdb1003 /srv partition
  • 20:15 hoo: Ran "scap pull" on snapshot1001 after T177486 related tests
  • 20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: partial rollback -- wikidata errors
  • 20:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.11
  • 20:05 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
  • 19:32 hoo@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
  • 19:24 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2 (duration: 00m 38s)
  • 19:24 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
  • 19:23 hoo@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/maintenance/dispatchChanges.php: dispatch: track how long client selecting takes (duration: 00m 48s)
  • 19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id, take 2
  • 19:22 ppchelko@tin: Finished deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id (duration: 00m 11s)
  • 19:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@8c66189]: Deduplicate based on event sha1 as well as on id
  • 19:20 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id (duration: 00m 09s)
  • 19:19 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Deduplicate based on event sha1 as well as on id
  • 18:44 volans: stopped irc echo on tegmen, re-enabled puppet and run it (it was disabled by a run-no-puppet sync_icinga_state)
  • 18:32 mobrovac@tin: Finished deploy [zotero/translators@3044b3a]: Update translators to 092c7bc - T178596 (duration: 00m 07s)
  • 18:32 mobrovac@tin: Started deploy [zotero/translators@3044b3a]: Update translators to 092c7bc - T178596
  • 18:16 herron: upgrading rhodium to puppet 4
  • 18:13 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 2/2 - T182023 (duration: 00m 48s)
  • 18:12 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch htmlCacheUpdate jobs for wiktionaries to EventBus, file 1/2 - T182023 (duration: 00m 48s)
  • 18:05 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 49s)
  • 17:51 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023 (duration: 00m 32s)
  • 17:50 ppchelko@tin: Started deploy [cpjobqueue/deploy@df72b34]: Switch htmlCacheUpdate for wiktionaries, attempt 2 T182023
  • 17:44 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023 (duration: 02m 57s)
  • 17:42 ppchelko@tin: Started deploy [cpjobqueue/deploy@3281df1]: Switch htmlCacheUpdate for wiktionaries T182023
  • 17:36 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1260.eqiad.wmnet
  • 17:05 hoo: Started Wikidata RDF dumps (sudo -b -u datasets bash -c 'dumpwikidatardf.sh all ttl; dumpwikidatardf.sh truthy nt') on snapshot1007
  • 16:33 anomie@terbium: Finished cleanupUsersWithNoId.php on mediawikiwiki
  • 16:29 anomie@terbium: Running cleanupUsersWithNoId.php for mediawikiwiki, see T181731
  • 16:28 anomie@terbium: Finished cleanupUsersWithNoId.php on testwikidatawiki
  • 16:27 anomie@terbium: Running cleanupUsersWithNoId.php for testwikidatawiki, see T181731
  • 16:26 anomie@terbium: Finished cleanupUsersWithNoId.php on test2wiki
  • 16:14 anomie@terbium: Running cleanupUsersWithNoId.php for test2wiki, see T181731
  • 16:11 anomie@terbium: Finished cleanupUsersWithNoId.php on testwiki
  • 16:03 anomie@terbium: Running cleanupUsersWithNoId.php for testwiki, see T181731
  • 15:13 akosiaris: upload kubernetes_1.7.10-1_amd64 on apt.wikimedia.org/stretch-wikimedia/main T181489
  • 14:59 addshore: swat done
  • 14:57 addshore@tin: Synchronized php-1.31.0-wmf.11/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
  • 14:53 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase/repo/includes/ChangeDispatcher.php: SWAT Tracking within ChangeDispatcher::getPendingChanges (duration: 00m 49s)
  • 14:33 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://studiezaal.nijmegen.nl to $wgCopyUploadsDomains (T181713) (duration: 00m 47s)
  • 14:22 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Portal namespace for mwl.wikipedia (T180052) (duration: 00m 48s)
  • 14:13 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180476 wmgUseNewWikiDiff2Extension true for dewiki (duration: 00m 48s)
  • 14:08 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T181498 T181329 Enable AdvancedSearch on fawiki and huwiki (duration: 00m 50s)
  • 13:40 moritzm: reimaging mw1260 to stretch
  • 10:55 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
  • 10:55 moritzm: reimaging mw2119 to stretch
  • 10:45 mobrovac@tin: Finished deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801 (duration: 06m 02s)
  • 10:39 mobrovac@tin: Started deploy [restbase/deploy@b1d7c82]: Use Cass3 for revisions, deprecate trending-edits, fix CX end point - T179421 T180384 T173801
  • 09:34 gehel: shuttting down logstash / elasticsearch on logstash100[123] in preparation for decommission -T175830
  • 08:05 moritzm: reimaging mw2118 to stretch (now for real, yesterday's reimage logged to SAL was interrupted)
  • 07:33 bblack: cp4024 - backend restart, mailbox lag
  • 02:39 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 08m 53s)
  • 01:01 eileen: update process-control to 55529ea - threshold for $500+ dedupe was too low
  • 00:36 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable description usage tracking for all wikis except commons T106287 (duration: 00m 48s)
  • 00:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on wikis with zero high priority linter errors T182042 (duration: 00m 51s)

2017-12-05

  • 23:07 eileen: update civicrm from d2c70d2 to 85263f6 (Major gifts address - choose latest)
  • 22:29 eileen: update process_control to 3522186 renable jobs to fetch recipient data
  • 22:13 mutante: restbase1010 failed at reboot with P6431 , after a cold start (power off, power on) it came back though :) (T178177 T141756)
  • 22:00 mutante: restbase1010 - rebooting for firmware upgrade
  • 21:58 mutante: restbase1010 - upgraded HP firmware (Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]) T141756 T178177
  • 21:56 urandom: draining cassandra instances, restbase1010 - T178177
  • 21:52 mutante: restbase1010 - upgrading firmware - Flashing Smart Array P440ar in Slot 0 [ 3.56 -> 6.06 ]
  • 21:31 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 shenanigans / wmf.11
  • 21:12 demon@tin: Finished scap: bootstrap wmf.11 (duration: 51m 01s)
  • 20:50 twentyafterfour: phabricator: running `sudo bin/garbage collect --collector search.ferret.ngram`
  • 20:29 twentyafterfour: phabricator: running `sudo bin/search ngrams --threshold 0.2`
  • 20:21 demon@tin: Started scap: bootstrap wmf.11
  • 19:50 mutante: forcing puppet ron on ores eqiad to reduce number of celery workers used for stress test
  • 19:37 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017 (duration: 01m 29s)
  • 19:36 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. T180017
  • 19:20 gehel: moving eventlogging collection by logstash from logstash1003 to logstash1007, no messages **should** be lost - T175830
  • 18:24 ppchelko@tin: Finished deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017 (duration: 00m 14s)
  • 18:24 ppchelko@tin: Started deploy [eventlogging/eventbus@6ca0372]: Make the kafka async deliver callback thread-safe. Limited to kafka1001. T180017
  • 17:27 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2118.codfw.wmnet
  • 15:56 awight: beginning stress test on ores* (non-production)
  • 15:54 awight@tin: Finished deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2] (duration: 01m 01s)
  • 15:53 awight@tin: Started deploy [ores/deploy@6baed71]: (non-production) Test ORES deployment to ores100[1-2]
  • 15:15 ottomata: restarrting kafka-jumbo brokers, applying SSL (downtime scheduled)
  • 15:03 urandom: bootstrapping cassandra, restbase1014-c - T179422
  • 14:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 43s)
  • 14:41 zeljkof: EU SWAT finished
  • 14:40 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Add category collation for sewiki" (T181503) (duration: 00m 44s)
  • 14:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHTML on itwiki and dewiki (T181188 T181190) (duration: 00m 43s)
  • 13:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight (duration: 00m 44s)
  • 13:14 moritzm: reimaging mw2118/mw2119 (video scalers) to stretch
  • 12:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully repool db1034 (duration: 00m 43s)
  • 12:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 44s)
  • 11:58 marostegui: Upgrade MariaDB and kernel on db1076
  • 11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)
  • 10:53 moritzm: rebooting meitnerium/archiva.wikimedia.org for update to 4.9.51
  • 10:46 moritzm: rebooting einsteinium (icinga.wikimedia.org) for update to 4.9,51
  • 10:45 elukey: reboot druid1003 for kernel+jvm updates - T179943
  • 10:33 addshore@tin: Synchronized wmf-config/extension-list: extension-list extension.json entrypoint for ArticlePlaceholder (duration: 00m 43s)
  • 10:33 moritzm: rebooting tegmen for update to 4.9,51
  • 10:07 jynus: stopping dbstore1002 (s5) and dbstore2001 (s5) for maintenance
  • 10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)
  • 10:00 ema: cp4021: restart varnish-be due to mbox lag/fetch failures
  • 09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359! (duration: 00m 43s)
  • 09:42 elukey: reboot analytics100[12] for kernel+jvm updates (Hadoop Master nodes) - T179943
  • 09:39 godog: bootstrap restbase1014-b - T179422
  • 09:20 marostegui: Optimize s7 on db1098 - T178359
  • 09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174569 (duration: 00m 43s)
  • 09:01 marostegui: Deploy schema change on db1076 (s2) - T174569
  • 08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174569 (duration: 00m 44s)
  • 08:54 moritzm: enabling test production traffic for mw1259 (stretch-based video scaler)
  • 08:24 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
  • 08:08 moritzm: installing python updates on trusty
  • 07:27 kartik@tin: Finished deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf (duration: 03m 22s)
  • 07:23 kartik@tin: Started deploy [cxserver/deploy@4b74f03]: Update cxserver to 1693bcf
  • 06:58 marostegui: Stop MySQL on db1034 to clone db1098:3317 - T178359
  • 06:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359! (duration: 00m 43s)
  • 06:48 marostegui: Fix dbstore1002 replication
  • 06:20 marostegui: Deploy schema change on db1090 (s2) - T174569
  • 06:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174569 (duration: 00m 43s)
  • 06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 - T174569 (duration: 00m 44s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 36s)
  • 00:29 mutante: bast4001 - removing ganglia aggregators, package, config...

2017-12-04

  • 23:44 mutante: bast3002 - killall -u ganglia to kill all aggregator procs, apt-get remove --purge ganglia-monitor, rm -rf /etc/ganglia, rm -rf /usr/lib/ganglia, apt-get autoremove
  • 23:28 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 44s)
  • 22:08 brion: brion running requeueTranscodes.php on terbium to batch-run .mp3 output on Commons (T181749)
  • 21:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided) (duration: 00m 42s)
  • 21:26 ppchelko@tin: Started deploy [cpjobqueue/deploy@c8bea2e]: (no justification provided)
  • 21:26 awight@tin: Finished deploy [ores/deploy@6baed71]: Update ORES to 6baed71 (duration: 12m 54s)
  • 21:25 eileen: re-enable dedupe job process control commit 461f482
  • 21:13 awight@tin: Started deploy [ores/deploy@6baed71]: Update ORES to 6baed71
  • 21:07 gehel@tin: Finished deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
  • 21:07 gehel@tin: Started deploy [kartotherian/deploy@6e223df]: testing new kartotherian packaging on maps-test2003
  • 20:07 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Revert wgCommentTableSchemaMigrationStage change, breaks too much stuff (duration: 00m 43s)
  • 19:31 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on test wikis (T166733) (duration: 00m 43s)
  • 19:18 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 03s)
  • 19:18 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
  • 19:17 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: Set $wgRestrictionMethod = 'firejail'; everywhere (T173370) (duration: 00m 43s)
  • 19:11 legoktm@tin: Synchronized static/images/mobile/copyright/: Add converted copyright svg images as png files - https://gerrit.wikimedia.org/r/#/c/394820/ (duration: 00m 43s)
  • 19:06 legoktm@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: Add ORES filter thresholds for simplewiki (duration: 00m 43s)
  • 18:46 hoo: Started dumpwikidatajson.sh on snapshot1007 (T181385)
  • 18:40 hoo: Ran scap pull on snapshot1001 (T181385)
  • 18:35 hoo@tin: Synchronized php-1.31.0-wmf.10/includes/objectcache/ObjectCache.php: Only send statsd data for WAN cache in non-CLI mode (T181385) (duration: 00m 44s)
  • 18:28 godog: bootstrap restbase1014-a - T179422
  • 18:04 gehel@tin: Finished deploy [wdqs/wdqs@2873745]: wdqs GUI update (duration: 02m 10s)
  • 18:02 gehel@tin: Started deploy [wdqs/wdqs@2873745]: wdqs GUI update
  • 17:41 demon@tin: Synchronized php-1.31.0-wmf.10/extensions/CirrusSearch/maintenance/: fix some deprecated spam (duration: 00m 44s)
  • 17:40 ejegg: disabled CiviCRM bounce processing again
  • 17:12 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 (duration: 02m 29s)
  • 17:08 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 (duration: 02m 58s)
  • 16:54 demon@tin: Synchronized docroot/noc/conf/dblists: double checking symlink move (duration: 00m 44s)
  • 16:44 ejegg: re-enabled CiviCRM bounced mail processing
  • 16:41 marostegui: Deploy schema change on db1053 (s2) - T174569
  • 16:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 - T174569 (duration: 00m 45s)
  • 16:39 demon@tin: Finished scap: docroot/noc/conf/ Adding new dblist symlink (duration: 02m 36s)
  • 16:39 marostegui: Deploy schema change on db1103:3312 - T174569
  • 16:37 demon@tin: Started scap: docroot/noc/conf/ Adding new dblist symlink
  • 16:37 demon@tin: scap aborted: docroot/noc/conf/dblists (duration: 00m 02s)
  • 16:37 demon@tin: Started scap: docroot/noc/conf/dblists
  • 16:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 - T174569 (duration: 00m 45s)
  • 16:32 demon@tin: Finished scap: docroot/noc/conf/ drop some dangling symlinks (duration: 03m 53s)
  • 16:28 demon@tin: Started scap: docroot/noc/conf/ drop some dangling symlinks
  • 16:11 herron: re-enabling puppet agents in eqiad
  • 16:07 demon@tin: Finished scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups (duration: 21m 19s)
  • 16:05 herron: re-enabling puppet agents in codfw
  • 15:46 demon@tin: Started scap: Revert "Special:Preferences: Use OOjs UI" and follow-ups
  • 15:31 herron: temporarily disabling puppet agents in eqiad and codfw while puppetdb catches up with command queue
  • 15:29 herron: disabling ircecho temporarily
  • 15:24 _joe_: restarting puppetdb on nihal, will cause puppet failures
  • 15:23 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1259.eqiad.wmnet
  • 15:18 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 45s)
  • 15:01 godog: reimage restbase1014 - T179422
  • 14:51 herron: cutting over all production puppet agents to codfw puppet 4 masters via dns
  • 14:50 Amir1: deployed backward compatibility of entity compact diff transmit T113468
  • 14:49 ladsgroup@tin: Synchronized php-1.31.0-wmf.10/extensions/Wikibase: (no justification provided) (duration: 01m 41s)
  • 14:37 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242 (duration: 00m 03s)
  • 14:37 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: dummy kartotherian deployment to test udp2log config change - T175242
  • 14:30 elukey: reboot druid100[23] for kernel updates
  • 14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Localize sitename and meta NS for wawiktionary (T181782) (duration: 00m 46s)
  • 14:01 elukey: reboot analytics106* (hadoop worker nodes) for kernel+jvm updates - T179943
  • 13:24 paravoid: upgrading bast2001 to stretch
  • 13:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 - T174569 (duration: 00m 45s)
  • 13:20 marostegui: Deploy schema change on db1105:3312 (s2) - T174569
  • 13:05 marostegui: Compress s6 on db1098 - T178359
  • 12:44 gehel@tin: Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 22s)
  • 12:43 gehel@tin: Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003
  • 11:06 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 10:37 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2085 (duration: 00m 43s)
  • 10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1096:3316 - T178359wq! (duration: 00m 45s)
  • 09:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1099:3318 from s5 (duration: 00m 44s)
  • 09:51 godog: bootstrap restbase1012-c - T179422
  • 09:32 godog: clear erroneous table metrics from graphite1003 / graphite2002 - T181689
  • 09:24 elukey: reboot analytics105* (hadoop worker nodes) for kernel+jvm updates - T179943
  • 09:19 jynus: rebooting mariadb at labsdb1005
  • 09:12 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (some controlled live tests following)
  • 08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and 3316 - T178359 (duration: 00m 45s)
  • 08:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3316 - T178359 (duration: 00m 45s)
  • 08:44 moritzm: updating tor on radium to 0.3.1.9
  • 08:41 moritzm: updating tor packages to 0.3.1.9
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and pool db1096:3316 - T178359 (duration: 00m 45s)
  • 08:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 - T178359 (duration: 00m 44s)
  • 08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1096:3315 - T178359 (duration: 00m 45s)
  • 07:53 moritzm: installing curl security updates
  • 07:17 marostegui: Compress s1 on db1099 - T178359
  • 07:08 marostegui: Stop MySQL on db1044 as it will be decommissioned - T181696
  • 07:05 _joe_: playing with puppetdb status for ores2003 (deactivating/reactivating node)
  • 06:40 marostegui: Stop MySQL on db1098 to clone db1096.s6 - T178359
  • 06:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
  • 06:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)
  • 06:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T178359 (duration: 00m 46s)
  • 06:21 marostegui: Deploy alter table on s3 master (db1075) without replication - T174569
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 28s)

2017-12-03

  • 15:33 ejegg: disabled CiviCRM bounce processing job
  • 12:17 akosiaris: empty ganeti1006, it had issues this morning per T181121
  • 12:06 marostegui: Fix dbstore1002 replication
  • 07:44 akosiaris: ran puppet on conf2002, etcdmirror-conftool-eqiad-wmnet got started again
  • 05:11 andrewbogott: deleting files on labsdb1003 /srv/tmp older than 30 days
  • 03:57 no_justification: gerrit2001: icinga is flapping on the gerrit process/systemd check, but this is kind of known (not sure why it's doing this all of a sudden). It's not letting me acknowledge it, but it's fine/harmless. Cf T176532

2017-12-02

  • 17:55 marostegui: Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359
  • 03:51 hoo: Ran "scap pull" on snapshot1001, after final T181385 tests
  • 00:03 mutante: tried one more time on db2028,db2029, both trusty. on db2028: gmond was running as user ganglia-monitor, failed, had to manually kill the process, run puppet again then ok. on db2029, gmond was running as "499" but puppet just ran and removed it without manual intervention. (T177225)

2017-12-01

  • 23:15 urandom: starting cassandra bootstrap, restbase1012-b - T179422
  • 21:49 mutante: db2029 - removing ganglia-monitor, testing to kill gmond, running puppet to figure out how to cleanly remove it on trusty
  • 21:12 mutante: db2023 killed gmond (ganglia-monitor) process manually which was still running even though ganglia-monitor package was removed and caused puppet breakage (it seems only on trusty). after that puppet run is clean again and ganglia removed. (T177225) (https://gerrit.wikimedia.org/r/#/c/394647/1)
  • 20:18 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores100*
  • 20:17 awight@tin: Finished deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001 (duration: 02m 31s)
  • 20:15 awight@tin: Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001
  • 20:03 aaron@tin: Synchronized php-1.31.0-wmf.10/includes/libs/objectcache/WANObjectCache.php: f096d0b465b75d - temp logging for statsd spam (duration: 00m 45s)
  • 18:59 demon@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 46s)
  • 18:22 mutante: Phabricator: restarting Apache for php-curl update
  • 18:21 _joe_: restarting apache2 on the codfw puppetmasters
  • 18:06 marktraceur@tin: Synchronized php-1.31.0-wmf.10/extensions/UploadWizard/resources/controller/uw.controller.Deed.js: (no justification provided) (duration: 00m 46s)
  • 17:49 mutante: phab2001 - restarted apache
  • 17:33 herron: stopped ircecho on einsteinium
  • 17:00 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 14s)
  • 17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 16666666666m 39s)
  • 17:00 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: -1m 59s)
  • 16:59 awight@tin: Unlocked for deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (duration: 00m 07s)
  • 16:59 awight@tin: Locking from deployment [ores/deploy]: Don't deploy while we're messing with git-lfs (planned duration: 60m 00s)
  • 16:34 jynus: stopping db2092 to clone s1 to db2085
  • 16:24 urandom: starting cassandra bootstrap, restbase1012-a -- T179422
  • 15:27 godog: bounce uwsgi on labmon1001 - stuck
  • 15:21 moritzm: installing nspr security updates on trusty
  • 15:17 moritzm: installing ffmpeg security updates
  • 15:14 gehel@tin: Finished deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003 (duration: 00m 20s)
  • 15:14 jynus@tin: Synchronized wmf-config/db-codfw.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
  • 15:14 gehel@tin: Started deploy [kartotherian/deploy@df7ebff]: testing new kartotherian packaging on maps-test2003
  • 15:14 moritzm: installing libxcursor security updates on trusty
  • 15:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Undeploy db2092, use db2085 for s1 (duration: 00m 45s)
  • 14:24 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 02m 06s)
  • 14:22 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
  • 14:22 akosiaris: upload apertium-crh-tur_0.3.0~r83159-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
  • 14:10 herron: cutting all puppet service records over to codfw puppet 4 masters
  • 12:44 elukey: reboot druid1001 for kernel+jvm updates - T179943
  • 12:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 00m 45s)
  • 11:59 jynus: restarting and upgrading mysql on labsdb1004
  • 11:28 jynus: upgrading and restarting dbstore2001
  • 11:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2055 (duration: 00m 46s)
  • 11:08 marostegui: Stop MySQL on db2055 for testing
  • 11:04 marostegui: Update MySQL on db1039 for testing
  • 10:57 elukey: reboot analytics1028 for kernel + jvm updates (Hadoop HDFS journalnode) - T179943
  • 10:56 godog: delete docker diskspace metrics from labs - T181476
  • 10:21 godog: initial purge of old table metrics from graphite2002 - T181689
  • 10:18 moritzm: reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (with some live tests starting next week)
  • 09:55 akosiaris: run memtester 61G on ganeti1008 T181121
  • 09:23 elukey: reboot analytics104* for kernel+jvm updates - T179943
  • 09:23 akosiaris: powercycle ganeti1008 T181121, it's largely unresponsive
  • 08:41 marostegui: Remove db1046 and db1047 from tendril - T156844
  • 08:40 elukey: reboot the remaining analytics103* hadoop workers to pick up kernel+jvm updates - T179943
  • 08:28 akosiaris: empty ganeti1008, move VMs to ganeti1006 T181121
  • 08:20 akosiaris: repool ganeti1005, ganeti1006 to empty ganeti1008 T181121
  • 07:51 akosiaris: upload apertium-tur_0.2.0~r83161-1+wmf1, apertium-crh_0.2.0~r83161-1+wmf1 to apt.wikimedia.org/jessie-wikimedia component main. T181465
  • 07:14 marostegui: Logging retroactively for the record, restarting MySQL on db1039
  • 05:14 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 44s)
  • 01:46 hoo: Ran scap pull on mwdebug1001 after T181385 testing
  • 00:13 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: HTML5 sections be upon us! (duration: 00m 45s)
  • 00:04 hoo: Killed all remaining Wikidata JSON/RDF dumpers, due to T181385. This means no dumps this week!

2017-11-30

  • 23:54 demon@tin: Synchronized wmf-config/extension-list-labs: no-op (duration: 00m 45s)
  • 23:39 addshore@tin: Synchronized wmf-config: wdbuild: T173818 T177060 Add wikidata extensions to extension-list (duration: 00m 46s)
  • 23:35 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 46s)
  • 23:34 addshore@tin: Synchronized wmf-config/Wikibase.php: wdbuild: T173818 Remove Wikibase-buildentry.php config file (empty) (duration: 00m 45s)
  • 23:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 46s)
  • 23:29 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: wdbuild: T173818 Remove wmgUseWikidataBuild (duration: 00m 45s)
  • 23:21 addshore@tin: Synchronized wmf-config: wdbuild: T173818 wdbuild: Stop loading from build on ALL WIKIS The build is dead! Mwahahaaa (duration: 00m 47s)
  • 23:18 bsitzmann@tin: Finished deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 (T181743) (duration: 04m 50s)
  • 23:15 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on all wikis (except the one i really dont want to break) (duration: 00m 46s)
  • 23:14 bsitzmann@tin: Started deploy [mobileapps/deploy@4305d96]: Update mobileapps to 4317ea5 (T181743)
  • 23:06 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group1 (duration: 00m 46s)
  • 22:55 addshore@tin: Synchronized wmf-config: wdbuild: wdbuild: Stop loading from build on group0 (duration: 00m 46s)
  • 22:42 addshore: BETA ONLY was a lie on that last one ...
  • 22:41 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY wdbuild: Stop loading from build on test and testwikidata (duration: 00m 47s)
  • 22:40 demon@tin: Pruned MediaWiki: 1.31.0-wmf.8 [keeping static files] (duration: 02m 33s)
  • 22:27 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY gerrit:394411 (duration: 00m 47s)
  • 21:35 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
  • 21:31 jynus: upgrading labsdb1010 and restarting mariadb
  • 21:23 mutante: powercycling kafka1018 (was down in Icinga and saw in SAL: reboot kafka10[12-22] for kernel + jvm updates - T179943)
  • 21:18 jynus: upgrade and restart db2078
  • 21:14 addshore@tin: Synchronized wmf-config: wdbuild: BETA ONLY (duration: 00m 47s)
  • 21:01 addshore@tin: Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 46s)
  • 20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 45s)
  • 20:49 XenoRyet: Updated tools from 6e604fd to 626fe02
  • 20:40 bd808: Sending Toolforge survey final reminder emails from silver for T177126
  • 20:32 addshore@tin: Synchronized wmf-config: REVERT wdbuild: T173818: add switch to ease killing (duration: 00m 47s)
  • 20:30 addshore@tin: Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (duration: 00m 47s)
  • 20:13 ejegg: re-adjusted Civi job timings. QC every odd min for 105 sec, TY every even min for 70 sec after 45 sec delay
  • 20:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.10
  • 19:55 ejegg: 105 seconds for each job
  • 19:55 ejegg: adjusted timings of Civi jobs to let TY and QC run concurrently
  • 19:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Temporary disable remex html" T178632 (duration: 00m 49s)
  • 19:20 thcipriani@tin: Synchronized php-1.31.0-wmf.10/includes/htmlform/fields/HTMLMultiSelectField.php: SWAT: HTMLMultiSelectField: Allow formatting in section headings in OOUI mode T181698 (duration: 00m 49s)
  • 19:12 thcipriani@tin: Synchronized wmf-config: SWAT: Enable MP3 uploads on Commons T120288 (duration: 00m 51s)
  • 17:57 andrewbogott: upgrading labtestpuppetmaster2001 to puppet 4.8
  • 17:47 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511)
  • 17:06 herron: beginning cut over of esams to codfw puppet 4 masters
  • 16:44 urandom: drop (erroneous) legacy tables from -ng cassandra cluster - T181689
  • 16:12 elukey: drain and reboot analytics1031->39 to pick up jvm+kernel updates - T179943
  • 15:59 jynus: setup prometheus with unix_socket on new server db1107 and db1108
  • 15:19 gehel: restart blazegraph on wdqs1004
  • 15:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 48s)
  • 15:11 herron: beginning cut over of ulsfo to codfw puppet 4 masters
  • 14:50 zeljkof: EU SWAT finished
  • 14:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikiLove Extension on pa.wiki (T178919) (duration: 00m 49s)
  • 14:28 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to de.wiki (T181695) (duration: 00m 49s)
  • 14:20 zfilipin@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/modules/dashboard/ext.cx.dashboard.js: SWAT: Set default languages after fetching valid languages (duration: 00m 49s)
  • 14:19 marostegui: Deploy schema change on s3 - db1078 - T174569
  • 13:23 addshore@tin: Synchronized php-1.31.0-wmf.10/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394297 (duration: 00m 48s)
  • 13:22 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/modules/ext.advancedSearch.init.js: pre-swat: T181644 Force search profile advanced in AdvancedSearch gerrit:394299 (duration: 00m 50s)
  • 13:18 herron: cutting codfw puppet agents over to puppetmaster2001.codfw.wmnet
  • 12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Tackle s4 DB weights to make them more equal (duration: 00m 48s)
  • 11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1081 and reduce traffic for db1091 (duration: 00m 50s)
  • 10:33 jynus: stopping replication on db1044 and db1095 (s3)
  • 09:16 moritzm: installing exim security updates on stretch (jessie/trusty not affected)
  • 09:14 elukey: drain and reboot analytics1029/1030 for jvm+kernel updates (Hadoop worker canaries)
  • 09:05 godog: add 200G of space to graphite2002 carbon lv
  • 08:25 moritzm: rolling restart of mw canaries to pick up curl security update
  • 08:21 marostegui: Enable GTID on s8 eqiad hosts that do not have it enabled (db1109, db1104, db1101, db1092, db1087, db1063) - T177208
  • 07:48 moritzm: installing curl security updates
  • 07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 18s)
  • 06:48 marostegui: Deploy schema change on dbstore1001 s3 - T174569
  • 06:26 marostegui: Deploy schema change on s3 db1077 - T174569
  • 02:48 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 03m 15s)
  • 02:45 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 08m 43s)
  • 01:17 ejegg: updated fundraising tools from 6d4b6f3 to 6e604fd
  • 01:06 twentyafterfour: finished phabricator upgrade, service is online and appears to be functioning normally
  • 01:03 twentyafterfour: Starting phabricator upgrade & maintenance. Service will be offline for less than 5 minutes.
  • 01:02 bsitzmann@tin: Finished deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004) (duration: 06m 08s)
  • 00:56 bsitzmann@tin: Started deploy [mobileapps/deploy@fa2a877]: Update mobileapps to dcea7d3 (T181004)
  • 00:53 maxsem@tin: Finished scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/ (duration: 26m 38s)
  • 00:48 ejegg: re-enabled CiviCRM jobs
  • 00:42 ejegg: updated CiviCRM from 0f95f3e to e81228f
  • 00:40 ejegg: disabled CiviCRM jobs
  • 00:31 ejegg: updated fundraising dashboard from 6ee6567 to 1141317
  • 00:27 maxsem@tin: Started scap: Message updates for https://gerrit.wikimedia.org/r/#/c/394155/
  • 00:24 maxsem@tin: Synchronized php-1.31.0-wmf.10/extensions/ContentTranslation/: https://gerrit.wikimedia.org/r/#/c/394206/ (duration: 00m 52s)
  • 00:21 ejegg: added weekly Ingenico audit processing job in makemissing mode

2017-11-29

  • 23:20 eileen: update civicrm from a1022cf to 0f95f3e (Benevity import fix)
  • 22:23 hashar: Nodepool had some troubles spawning new instances from 21:09 to 21:36, and took a while to recover. Issue similar to T170492#3581822
  • 21:51 SMalyshev: starting wikidata reindex (T181426)
  • 21:23 no_justification: docroot sync was for If1afa59a
  • 21:21 demon@tin: Synchronized docroot/: (no justification provided) (duration: 00m 49s)
  • 20:29 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10
  • 20:26 demon@tin: Synchronized php: symlink bump (duration: 00m 48s)
  • 20:05 urandom: restarting cassandra bootstrap of restbase1012-a (T179422)
  • 19:55 mutante: restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865)
  • 19:50 herron: beginning rolling cut over of eqiad scb hosts to codfw puppet 4 masters
  • 19:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/394059/3 (duration: 00m 49s)
  • 18:57 herron: beginning cutover of codfw lvs2* systems to codfw puppet 4 masters. first standby nodes, then active nodes
  • 18:12 marostegui: Compress s5 on db1096 - T178359
  • 18:10 jynus@tin: Synchronized wmf-config/db-eqiad.php: Increase db1110 load (duration: 00m 53s)
  • 18:05 awight@tin: Finished deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster (duration: 01m 11s)
  • 18:04 awight@tin: Started deploy [ores/deploy@532bd0b]: (non-production) Update ORES on new cluster
  • 18:03 herron: beginning cut over of codfw cp2* servers to codfw puppet 4 masters
  • 17:40 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1110 with low load (duration: 00m 48s)
  • 17:40 godog: bootstrapping restbase1012-a - T179422
  • 17:19 herron: beginning cut over of cp200[12] to codfw puppet 4 masters
  • 17:16 ejegg: increased time limit for donation queue consumer from 60 to 70 seconds, reduced thank you job from 55 to 45 seconds
  • 17:03 ejegg: re-enabled Civi jobs
  • 16:58 ejegg: disabled Civi jobs for Deadlock-retry update
  • 16:46 jynus@tin: Synchronized wmf-config/db-codfw.php: Update db1110 ip (duration: 00m 50s)
  • 16:43 jynus@tin: Synchronized wmf-config/db-eqiad.php: Update db1110 ip (duration: 00m 49s)
  • 15:02 chasemp: purge old qcow2 images from under /home on labtestcontrol2001
  • 15:00 awight: Restarting ORES celery workers manually, T181538
  • 14:39 _joe_: reloading apache on puppetmasters to pick up the configuration changes
  • 14:37 awight@tin: Started restart [ores/deploy@e58bfbf]: Restart ORES services (take 2), T181538
  • 14:36 elukey: reboot druid100[456] for jvm+kernel updates - T179943
  • 14:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538 (duration: 00m 16s)
  • 14:35 awight@tin: Started deploy [ores/deploy@e58bfbf]: Restart ORES services, T181538
  • 14:25 zeljkof: EU SWAT finished
  • 14:18 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Define throttle rule (T181367) (duration: 00m 49s)
  • 14:14 herron: beginning cut over of cache::misc and codfw cache::canary cp servers to codfw puppet4 masters
  • 14:13 moritzm: rebooting neodymium for kernel update to 4.9.51
  • 14:12 godog: reimage restbase1012 - T179422
  • 14:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180291 T180128 AdvancedSearch for arwiki (duration: 00m 49s)
  • 14:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT T180128 AdvancedSearch for dewiki (duration: 00m 50s)
  • 13:34 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3) (duration: 16m 49s)
  • 13:18 elukey: reboot kafka100[23] for jvm+kernel updates - T179943
  • 13:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 3)
  • 12:34 jynus@tin: Synchronized wmf-config/db-eqiad.php: Setup s8 on eqiad, with no wikis (duration: 00m 48s)
  • 12:24 jynus: scap pull on mwdebug1001
  • 12:23 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db1096:3315 (duration: 00m 48s)
  • 12:01 marostegui: Deploy schema change on db1072 (sanitarium master) on s3 with replication enabled to replicate to labs - T174569
  • 11:30 elukey: reboot kafka1001 for kernel + jvm updates - T179943
  • 10:15 akosiaris: disable puppet on oresrdb* for merging https://gerrit.wikimedia.org/r/#/c/394022/. T181563
  • 09:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 48s)
  • 09:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 49s)
  • 09:39 godog: upload cassandra-tools-wmf 1.0.2-1 - T181438
  • 09:28 mobrovac@tin: Finished deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200 (duration: 04m 03s)
  • 09:24 mobrovac@tin: Started deploy [electron-render/deploy@94d27d7]: Update to electron v1.7.9 and start using the Charter font - T181200
  • 09:04 godog: bootstrap restbase1007-c - T179422
  • 08:41 moritzm: uploaded prometheus-openldap-exporter 0+git20171128-1 for jessie-wikimedia (T181511)
  • 08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3318 and db1055 - T178359 (duration: 00m 48s)
  • 07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 traffic - T178359 (duration: 00m 45s)
  • 07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099:3318 with low weight - T178359 (duration: 00m 45s)
  • 06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1055 - T178359 (duration: 00m 48s)
  • 06:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 48s)
  • 06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099:3311 and db1099:3318 to the config (depooled) T178359 (duration: 00m 49s)
  • 06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with lower weight - T178359 (duration: 00m 50s)
  • 06:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 04m 27s)
  • 06:12 ebernhardson@tin: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
  • 03:39 ejegg: reduced CiviMail sample rate to 0.35
  • 03:09 ejegg: reduced donation queue consumer time limit from 70 to 60 seconds, increased ty mail batch time limit from 45 to 55 seconds
  • 02:50 ejegg: reduced donation queue consumer time limit from 75 to 70 seconds, increased ty mail batch time limit from 40 to 45 seconds
  • 02:44 ejegg: reduced CiviMail record creation rate from 100% to 50%
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 06m 08s)
  • 01:25 mutante: snapshot1001 - closed idle screen session
  • 01:22 mutante: analytics1003 - closed idle screen session
  • 01:21 mutante: mw1276 - run "scap pull" to get in sync after hardware issue, then pooled again (T181397)
  • 01:09 mutante: restarting ircecho - it stopped talking
  • 01:07 mutante: forcing puppet run on all labvirt* machines to clean out Icinga alerts
  • 00:44 awight@tin: Synchronized php-1.31.0-wmf.10/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 49s)
  • 00:33 reedy@tin: Synchronized php-1.31.0-wmf.10/includes/logging/LogPager.php: Fix fatal on Special:Log T181565 (duration: 00m 48s)
  • 00:32 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: Hotfix to mitigate cache stampeding, T181567 (duration: 00m 50s)
  • 00:18 Jamesofur: deleted 6 archived files from servers for legal compliance
  • 00:08 ejegg: updated fundrasing dashboard from df94248 to 6ee6567

2017-11-28

  • 23:58 hoo: Ran scap pull on mwdebug1001 after T181385 related testing
  • 23:07 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: T181538 (duration: 00m 49s)
  • 22:31 akosiaris@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 49s)
  • 22:31 akosiaris: deploy wmf-config/CommonSettings.php for ORES internal discovery URL, https://gerrit.wikimedia.org/r/#/c/393924/ T181538
  • 21:41 akosiaris: disable ORES queue redis persistency by config set appendonly no on oresrdb1001
  • 21:24 hoo: Manually killed all remaining Wikidata TTL (RDF) dumpers on snapshot1007. Some shards failed due to the db1110 depool.
  • 21:23 hoo: Manually killed all remaining Wikidata JSON dumpers on snapshot1007. Some shards failed due to the db1110 depool.
  • 20:56 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: removing aawiki from group0
  • 20:42 gehel: repooling elastic2004 after RAID controller maintenance - T181412
  • 20:42 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10
  • 20:28 mutante: forcing puppet run on cache misc to revert "failover ORES to codfw"
  • 20:19 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 48s)
  • 20:16 demon@tin: Synchronized dblists/group0.dblist: adding some new wikis (duration: 00m 48s)
  • 18:57 demon@tin: Finished scap: bootstrap wmf.10 (duration: 35m 20s)
  • 18:51 demon@tin: Finished deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes (duration: 00m 09s)
  • 18:51 demon@tin: Started deploy [gerrit/gerrit@571cf4c]: deploying 2.15+ polygerrit style changes
  • 18:40 akosiaris: revert weight changes for scb1001, scb1002 T181835
  • 18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 18:39 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 18:35 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2) (duration: 04m 30s)
  • 18:32 akosiaris: force puppet run on cache::misc boxes T181538
  • 18:30 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (take 2)
  • 18:28 urandom: (re)bootstrapping cassandra, restbase1007-b - T179422
  • 18:22 demon@tin: Started scap: bootstrap wmf.10
  • 18:18 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster (duration: 01m 41s)
  • 18:17 awight@tin: Started deploy [ores/deploy@e58bfbf]: (non-production) Update ORES on new cluster
  • 18:00 akosiaris: force stop celery-ores-worker on scb1001
  • 17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 17:58 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
  • 17:46 urandom: decommissioning cassandra, restbase1007-b - T179422
  • 17:42 urandom: restart cassandra, restbase1007, to pickup logstash java deps - T179422
  • 16:45 ejegg: updated fundraising dashboard from d8c86e7 to df94248
  • 16:25 bblack: mw1329 boot to PXE (should come up with new .66 IP)
  • 15:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1110 (duration: 00m 44s)
  • 15:38 bblack: powered off mw1329
  • 15:06 marostegui: Compress s5 on db1099
  • 15:04 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Comply wikidata with new ores thresholds" (duration: 00m 45s)
  • 14:56 marostegui: Deploy schema change on s3 on dbstore1002 - T174569
  • 14:45 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 02s)
  • 14:45 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
  • 14:41 otto@tin: Finished deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440 (duration: 00m 02s)
  • 14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used T178440
  • 14:40 otto@tin: Started deploy [eventlogging/analytics@c464b8c]: Fixing bug where userAgent set by client producer was not used
  • 14:39 gehel@tin: Finished deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003 (duration: 00m 18s)
  • 14:39 gehel@tin: Started deploy [kartotherian/deploy@cb9b1ef]: testing new kartotherian packaging on maps-test2003
  • 14:37 marostegui: Stop MySQL on db1055 to clone db1099:3311 - T178359
  • 14:34 gehel@tin: Finished deploy [kartotherian/deploy@55c5da4]: (no justification provided) (duration: 00m 11s)
  • 14:34 gehel@tin: Started deploy [kartotherian/deploy@55c5da4]: (no justification provided)
  • 14:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 44s)
  • 14:17 elukey: reboot kafka10[12-22] for kernel + jvm updates - T179943
  • 14:03 elukey: reboot kafka200[123] for kernel + jvm updates - T179943
  • 12:25 godog: cleanup wanobjectcache metrics with hashes - T178531
  • 12:14 godog: bounce carbon-frontend-relay after https://gerrit.wikimedia.org/r/393749
  • 11:25 hoo: Manually re-started Wikidata JSON dumps on snapshot1007, got stuck after db1082 went down.
  • 10:55 jynus: stop db1044 replication for db1095 master switchover
  • 10:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082. Depool db1044 (duration: 00m 44s)
  • 10:44 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Process wikibase-addUsagesForPage only via EventBus - T175212 (duration: 00m 44s)
  • 10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency (duration: 00m 29s)
  • 10:07 ppchelko@tin: Started deploy [cpjobqueue/deploy@c4b9e16]: Enable wikibase-addUsagesForPage with low concurrency
  • 09:35 jynus: restart db1087 for maintenance
  • 09:22 jynus: restart db1082
  • 09:19 godog: unmask and restart restbase1007-b - T179422
  • 09:15 moritzm: installibg libxml-libxml-perl security updates on trusty (Debian already fixed)
  • 09:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming (duration: 00m 28s)
  • 09:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@b0b1793]: Remove double-processing created for consumer group renaming
  • 09:04 marostegui: Drop database log from dbstore1002 - T156844
  • 08:57 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml (duration: 00m 39s)
  • 08:56 ppchelko@tin: Started deploy [cpjobqueue/deploy@2212086]: Move enabled jobs config to vars.yaml
  • 08:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1082 (duration: 00m 44s)
  • 08:02 mobrovac: bootstrap restbase1007-b - T179422
  • 06:39 marostegui: Stop MySQL on db1099 to copy its content to dbstore1001 and reimage it as multi-instance - T178359
  • 05:23 mutante: labtestpuppetmaster2001 - manually purging ganglia things since puppet is disabled
  • 05:05 mutante: labweb1001/labweb1002: manually purging ganglia package/config/service/unit files because puppet is disabled there (T177225)
  • 04:08 eileen: update process-control to b40eaec, disabling 3 jobs to reduce load for Big english starting: dedupe_civicrm_contacts, omnimail_recipient_load, omnimail_recipient_load_backfill
  • 03:58 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 (duration: 03m 13s)
  • 03:31 mutante: labservices1001/1002,labtestservices2001 - remove pdns_gmetric cronjobs causing cron spam after ganglia decom from lab*
  • 02:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 23s)

2017-11-27

  • 22:29 mutante: labtestnet2001 - re-enabled puppet to decom ganglia, no errors
  • 22:22 mutante: decom'ing Ganglia from all labtest* hosts - packages removed by puppet etc, might cause some false positives in Icinga but it's ok (T177225)
  • 22:21 otto@tin: Finished deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab (we caught one). T180017 (duration: 00m 10s)
  • 22:21 otto@tin: Started deploy [eventlogging/eventbus@e024af3]: deploy revert checked out at 3df06ab (we caught one). T180017
  • 21:59 awight: clearing ORES threshold caches for ruwiki, frwiki, wikidatawiki.
  • 21:59 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Reenable ORES on frwiki, ruwiki, and wikidata; T181006 (duration: 00m 45s)
  • 21:53 awight@tin: Finished deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 05m 06s)
  • 21:47 awight@tin: Started deploy [ores/deploy@e58bfbf]: (codfw) Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
  • 21:46 awight@tin: Finished deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450 (duration: 12m 55s)
  • 21:33 awight@tin: Started deploy [ores/deploy@e58bfbf]: Update ORES for impossible threshold handling and new wikidata editquality model, T179711 T180686 T180450
  • 21:23 awight@tin: Synchronized php-1.31.0-wmf.8/extensions/ORES: ORES error handling for bad thresholds, T181191 (duration: 00m 46s)
  • 21:11 awight@tin: Synchronized wmf-config/InitialiseSettings.php: Temporarily disable ORES on wikidata (duration: 00m 45s)
  • 21:10 awight: Previous “rollback ORES” was from a stale screen session, no actions were taken today.
  • 21:08 awight@tin: Finished deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006 (duration: 9942m 42s)
  • 20:21 demon@tin: i lied, that was for wgStyleVersion removal
  • 20:20 demon@tin: Synchronized wmf-config/CommonSettings.php: cli sapi fixes (duration: 00m 45s)
  • 20:17 ejegg: updated payments-wiki from fea6b37 to f594dfa
  • 20:03 kaldari@tin: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 45s)
  • 20:02 kaldari@tin: Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 45s)
  • 20:01 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 46s)
  • 19:54 bblack: crN-ulsfo: remove lvs400[1-4] from PyBal BGP neighbors list
  • 19:42 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T181100 (duration: 00m 45s)
  • 19:41 demon@tin: Synchronized wmf-config/InitialiseSettings.php: babel officewiki thing (duration: 00m 45s)
  • 19:39 catrope@tin: Synchronized php-1.31.0-wmf.8/includes/specials/SpecialRecentchangeslinked.php: T181100 (duration: 00m 45s)
  • 19:34 otto@tin: Finished deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017 (duration: 00m 12s)
  • 19:34 otto@tin: Started deploy [eventlogging/eventbus@3df06ab]: Temp deploy to kafka1001 only to catch bug: T180017
  • 19:21 catrope@tin: Synchronized wmf-config/Wikibase.php: T176903 (duration: 00m 45s)
  • 19:17 AaronSchulz: Removed more bogus md5 wanobjecache/ metric from graphite[12]001
  • 19:11 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: T179241 (duration: 00m 45s)
  • 19:02 volans: restarted ircecho
  • 18:51 ejegg: updated payments-wiki from 6b3019b to fea6b37
  • 18:50 dcausse: elastic/cirrus: reindexing english group2 wikis: T179945
  • 18:45 volans: stopped ircecho temporary to avoid spam
  • 18:14 XioNoX: pushing v6 gre firewall filter
  • 18:12 gehel@tin: Finished deploy [wdqs/wdqs@7ad20f3]: (no justification provided) (duration: 02m 10s)
  • 18:11 gehel: deploying blazegraph + GUI updates
  • 18:10 gehel@tin: Started deploy [wdqs/wdqs@7ad20f3]: (no justification provided)
  • 18:08 hoo: Killed truthy nt dumpers stuck at 100% CPU (from last week) on snapshot1007 (T181385)
  • 18:06 akosiaris: poweroff ganeti1005, ganeti1006 T181121
  • 18:05 bd808: Sending Toolforge survey 1 week reminder emails from silver for T177126
  • 16:07 bblack: lvs400x - puppet disabled for https://gerrit.wikimedia.org/r/#/c/393610
  • 15:52 herron: beginning canary cutover/deployment of codfw prometheus servers to codfw puppet 4 puppetmasters
  • 15:19 chasemp: disable puppet across cloud things for cleanup
  • 15:11 herron: beginning canary cutover/deployment of codfw elasticsearch servers to codfw puppet 4 puppetmasters
  • 14:22 zeljkof: EU SWAT finished
  • 14:19 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: IP cap lift for Semaine contributive 2017-2018 (T181360) (duration: 00m 45s)
  • 14:10 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch: SWAT AdvancedSearch T181175 T181222, adjust links in beta section & fix usability of mobile search (duration: 00m 46s)
  • 13:31 moritzm: installing nspr security updates on trusty
  • 13:22 elukey: remove eventlogging replication support (log database) from dbstore1002 - T156844
  • 13:17 moritzm: updating nginx on francium
  • 13:01 Pchelolo: stop cpjobqueue in eqiad for backlog accumulation
  • 12:44 Pchelolo: started parsoid linter script to generate load on cpjobqueue (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://commons.wikimedia.org/w/api.php)
  • 12:44 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393587 BETA wmgMonologChannels FileImporter => debug (duration: 01m 01s)
  • 12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1276.eqiad.wmnet
  • 12:35 moritzm: powercycling mw1276
  • 12:21 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393582 LABS ONLY Actually call wfLoadExtension for FileExporter & Importer on beta BETA (T181383) (duration: 00m 44s)
  • 12:14 marostegui: Stop replication on db1109 to test table rename for s5/s8 failover
  • 12:09 ppchelko@tin: Finished deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007 (duration: 01m 03s)
  • 12:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007
  • 11:47 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT2/2 (T181383) (duration: 00m 45s)
  • 11:46 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: gerrit:393577 LABS ONLY Enable FileImporter & FileExporter on BETA PT1/2 (T181383) (duration: 00m 45s)
  • 11:44 addshore@tin: Synchronized wmf-config/extension-list-labs: gerrit:393576 BETA ONLY Add FileExporter & FileImporter to extension-list-labs (duration: 00m 45s)
  • 11:27 hashar: contint1001 enable Icinga "Disks space" notification again. It is no more complaing about Docker partitions | ping mutante | T178454
  • 11:17 godog: bootstrap restbase1007-a - T179422
  • 11:05 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 45s)
  • 11:05 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:53 moritzm: installing postgresql-common security updates
  • 10:08 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive (duration: 00m 22s)
  • 10:08 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive
  • 09:42 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive (duration: 00m 48s)
  • 09:41 ppchelko@tin: Started deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive
  • 09:14 godog: reimage restbase1007 - T179422
  • 08:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1097:3314 - T178359 (duration: 00m 43s)
  • 08:40 moritzm: installing openjdk security updates on hadoop, druid and kafka clusters
  • 08:27 marostegui: Deploy schema change on dbstore1002 and dbstore1001 - T174569
  • 08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
  • 07:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s)
  • 07:10 marostegui: Stop MySQL on db1021 as it will be decommissioned - T181378
  • 06:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s)
  • 06:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s)
  • 06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1097:3314 with low weight - T178359 (duration: 00m 46s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 44s)

2017-11-25

  • 19:31 marostegui: Set 32:3 disk to offline on db1051
  • 16:09 kartik@tin: Finished deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2 (duration: 03m 29s)
  • 16:05 kartik@tin: Started deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2
  • 16:05 godog: unban statsd traffic from scb on graphite1001 - T181333
  • 15:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333 (duration: 00m 31s)
  • 15:45 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333
  • 15:37 volans: restarted statsd-proxy on graphite1001 (died during investigation) T181333
  • 14:34 godog: rolling restart of cxserver to alleviate metrics leak - T181333
  • 14:26 godog: restart cxserver on scb100[34] - T181333
  • 14:10 godog: roll-restart cpjobqueue to alleviate metrics leak - T181333
  • 13:40 godog: drop incoming statsd from scb to graphite1001 temporarily - T181333
  • 08:32 ariel@tin: Finished deploy [dumps/dumps@ec21673]: fix abstracts recombine job (duration: 00m 02s)
  • 08:32 ariel@tin: Started deploy [dumps/dumps@ec21673]: fix abstracts recombine job

2017-11-24

  • 13:52 moritzm: removing git packages from jessie-wikimedia/experimental (replaced by component/git)
  • 13:24 moritzm: installing openjpeg2 updates (original security already got installed after initial release, but there was a binNMU for amd64)
  • 13:17 marostegui: Stop replication on db1097 to reimport and recompress commonswiki.watchlist
  • 12:54 jynus: reenabling puppet on db1071
  • 12:50 jynus: resetting replication on es1011 for consistency with other replica sets
  • 12:40 jynus: setting up s8 topology on eqiad
  • 12:38 jynus: disable puppet on db1071 and stop local s5 heartbeat there
  • 12:32 reedy@tin: Synchronized docroot/mediawiki/keys/: Fixup keys (duration: 00m 45s)
  • 12:13 marostegui: Enable GTID on es2018 - T181293
  • 11:57 marostegui: Disable puppet on es2018 - T181293
  • 11:50 jynus@tin: Synchronized wmf-config/db-codfw.php: depool es2018 T181293 (duration: 00m 45s)
  • 11:48 marostegui: Reboot es2018 after full-upgrade - T181293
  • 11:25 marostegui: Restart mysql on es2018
  • 11:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: db2085:3318, db2086:3318 (duration: 00m 43s)
  • 11:24 jynus@tin: Synchronized wmf-config/db-codfw.php: Pool db2038, db2085:3318, db2086:3318 (duration: 00m 45s)
  • 10:55 marostegui: Restart MySQL on db2086 to move s5 to s8
  • 10:37 jynus: cancelling db2085 restart, only doing mysql:s5
  • 10:35 jynus: restarting db2085 (including both s5 and s3 instances)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool all future s8 slaves for a topology change - T177208 (duration: 00m 45s)
  • 10:22 moritzm: installing ca-cerfificates updates on trusty hosts
  • 09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1097:3315 and db1092 (duration: 00m 45s)
  • 09:50 jynus: restarting db2045
  • 09:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 db1097:3315 abd db1092 (duration: 00m 45s)
  • 08:49 marostegui: Stop MySQL on db1092
  • 08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101:3318 in s5 to warm it up and depool db1092 - T178359 T177208 (duration: 00m 45s)
  • 08:40 moritzm: installing java security updates on notebook* hosts
  • 08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
  • 08:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1097:3315 - T178359 (duration: 00m 45s)
  • 08:20 moritzm: installing java security updates on meitnerium
  • 08:15 moritzm: installing java security updates on stat1004
  • 08:14 hashar: restarting jenkins on contint1001 for a java update
  • 08:07 elukey: re-enabling piwik on bohrium (only VM running on ganeti1006 atm) after mysql tables restore completed
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 45s)
  • 06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Slowly pool db1101:3318 in s5 to warm it up - T178359 (duration: 00m 49s)
  • 00:12 reedy@tin: Synchronized docroot/mediawiki/keys/: Add Brian Wolff's key (duration: 00m 45s)

2017-11-23

  • 23:42 reedy@tin: Synchronized wmf-config/: Simplify variable assignment (duration: 00m 47s)
  • 23:25 reedy@tin: Synchronized composer.lock: (no justification provided) (duration: 00m 44s)
  • 23:24 reedy@tin: Synchronized composer.json: (no justification provided) (duration: 00m 45s)
  • 23:23 reedy@tin: Synchronized phpcs.xml: (no justification provided) (duration: 00m 45s)
  • 22:39 demon@tin: Synchronized wmf-config/: style fixes, no-op (duration: 00m 47s)
  • 22:19 demon@tin: Synchronized wmf-config/CommonSettings.php: no-op, moving stuff around (duration: 00m 47s)
  • 21:34 ariel@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw2251.codfw.wmnet
  • 21:15 apergos: rebooted mw2251 after unresponsive on mgmt console and no ping
  • 20:49 demon@tin: Synchronized static/images/project-logos/nowikimedia.png: logo update (duration: 02m 51s)
  • 17:29 akosiaris: set ganeti1006 as drained. ganeti1005 was already set. That will prevent scheduling VMs on those. T181121
  • 17:01 akosiaris: force powercycle on ganeti1006 T181121
  • 16:30 akosiaris: gnt-node failover ganeti1006
  • 15:50 jynus: disable puppet on db2023, db2038 for deployment whle transfer is ongoing
  • 15:23 moritzm: rebooting mwdebug* servers for update to 4.9.51
  • 14:10 marostegui: Stop Mysql on db1101.s5 to clone db1097 - T178359
  • 14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1053 in s2 as vslow to replace db1021 - T134476 (duration: 00m 45s)
  • 13:13 jynus: sutting down db2023 and db2038 for cloning
  • 12:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: GWToolset migratory config T87928 (duration: 00m 46s)
  • 12:40 moritzm: migrating instances off ganeti2001 / kernel reboot to 4.9.51
  • 12:10 moritzm: migrating instances off ganeti2002 / kernel reboot to 4.9.51
  • 11:52 moritzm: migrating instances off ganeti2003 / kernel reboot to 4.9.51
  • 11:37 kartik@tin: Finished deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0 to fix Youdao MT (duration: 03m 01s)
  • 11:35 moritzm: migrating instances off ganeti2004 / kernel reboot to 4.9.51
  • 11:35 moritzm: migrating instances off ganeti200 / kernel reboot to 4.9.51
  • 11:34 kartik@tin: Started deploy [cxserver/deploy@51b78ce]: T181209 Update cxserver to e8fe3f0 to fix Youdao MT
  • 11:23 moritzm: migrating instances off ganeti2005 / kernel reboot to 4.9.51
  • 11:15 moritzm: migrating instances off ganeti2006 / kernel reboot to 4.9.51
  • 11:06 moritzm: migrating instances off ganeti2007 / kernel reboot to 4.9.51
  • 10:57 moritzm: migrating instances off ganeti2008 / kernel reboot to 4.9.51
  • 10:07 moritzm: rebooting sarin for update to 4.9.51
  • 09:16 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Switchover s5 codfw master (duration: 00m 45s)
  • 08:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 (duration: 00m 44s)
  • 08:44 jynus: stopping and restarting db2052
  • 08:31 moritzm: installing bdb security updates on trusty
  • 08:29 jynus: starting switchover of db2023 to db2052
  • 08:21 marostegui: Stop MySQL on db1072 to MySQL upgrade and kernel upgrade
  • 08:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 (duration: 00m 45s)
  • 07:28 marostegui: Stop MySQL on db1021 and db1053 to clone db1053
  • 07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1021 - T134476 (duration: 00m 45s)
  • 07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1053 to s2 to replace db1021 as vslow, dump slave - T134476 (duration: 00m 45s)
  • 06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T180045 (duration: 00m 45s)
  • 06:55 marostegui: Drop index on ores_classification on s1 - T180045
  • 06:53 marostegui: Drop index on ores_classification on s2 - T180045
  • 06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T180045 (duration: 00m 45s)
  • 06:44 legoktm: legoktm@tin:~$ echo "https://www.mediawiki.org/keys/keys.html" | mwscript purgeList.php
  • 06:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 46s)
  • 05:58 kartik@tin: Finished deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0 (duration: 04m 13s)
  • 05:54 kartik@tin: Started deploy [cxserver/deploy@5b35ed5]: Update cxserver to e8fe3f0
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 27s)
  • 01:36 demon@tin: Synchronized wmf-config/InitialiseSettings.php: dropping education program from cswiki (duration: 00m 45s)
  • 01:29 demon@tin: Synchronized wmf-config/CommonSettings.php: Set $wgCentralAuthGlobalBlockInterwikiPrefix (duration: 00m 45s)
  • 01:26 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 44s)
  • 01:24 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
  • 01:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: abusefilter stuff (duration: 00m 45s)
  • 01:19 demon@tin: Synchronized wmf-config/CommonSettings.php: abusefilter stuff (duration: 00m 45s)
  • 01:05 demon@tin: Synchronized docroot/mediawiki/keys/: update formatting, docs, etc (duration: 00m 46s)

2017-11-22

  • 21:15 mutante: running puppet on logstash hosts to apply config change 392897 (add log4j filter)
  • 21:11 demon@tin: Synchronized docroot/noc/conf/index.php: favicon fix (duration: 00m 46s)
  • 20:35 demon@tin: Synchronized dblists/: now with more alphabeticalizedness (duration: 00m 45s)
  • 20:25 demon@tin: Synchronized dblists/Makefile: no-op (duration: 00m 45s)
  • 20:23 demon@tin: Synchronized tests/noc-conf/NOCDblistTest.php: no-op (duration: 00m 45s)
  • 20:19 demon@tin: Synchronized wmf-config/LabsServices.php: no-op (duration: 00m 45s)
  • 20:14 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adding national library of Israel to copy domains (duration: 00m 45s)
  • 20:13 otto@tin: Finished deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625 (duration: 00m 04s)
  • 20:13 otto@tin: Started deploy [eventlogging/analytics@57234e7]: no-op: removing now unneeded code that might accidentally serialize userAgent to json string: T179625
  • 20:09 demon@tin: Synchronized robots.txt: block a nasty bot 💔 (duration: 00m 44s)
  • 20:02 demon@tin: Synchronized multiversion/bin/expanddblist: fix param warning (duration: 00m 45s)
  • 20:00 mepps: updated civicrm from a16e566 to a1022cf
  • 19:58 demon@tin: Synchronized wmf-config/InitialiseSettings.php: comments (duration: 00m 45s)
  • 19:56 demon@tin: Synchronized multiversion/MWScript.php: assume aawiki for purgeUrls (duration: 00m 45s)
  • 19:51 demon@tin: Synchronized docroot/: removing old foundation docroot (duration: 00m 46s)
  • 19:49 urandom: starting cassandra cleanups, restbase-200{1,3,5}-a - T179422
  • 19:41 demon@tin: Synchronized w/extract2.php: removing old portal support (duration: 00m 45s)
  • 18:47 demon@tin: Synchronized dblists/closed.dblist: closed transitionteamwiki (duration: 00m 45s)
  • 18:34 demon@tin: Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 45s)
  • 18:32 demon@tin: Synchronized scap/plugins/updatewikiversions.py: minor fix (duration: 00m 45s)
  • 18:30 demon@tin: Pruned MediaWiki: 1.31.0-wmf.7 [keeping static files] (duration: 01m 46s)
  • 18:12 herron: re-enabling puppet agents after puppetdb postgres security updates
  • 18:09 moritzm: installing postgres security updates on nitrogen/puppetdb
  • 18:05 moritzm: installing postgres security updates on nihal/puppetdb
  • 18:02 herron: disabling puppet agents for puppetdb postgres security update
  • 17:30 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch/: fixing layout issues in timeless (duration: 00m 46s)
  • 16:54 mepps: updated payments-wiki from 1ca91b1 to 6b3019b
  • 16:45 chasemp: disable puppet accross labtest things
  • 16:34 marostegui: Compress s4 on db1097 - T178359
  • 16:28 herron: starting canary deploy/cutover of codfw scb hosts to codfw puppet 4 masters
  • 16:16 elukey: restart druid broker,coordinator,historical daemons on druid100[123] to pick up new logging settings
  • 15:41 jynus: starting manually pt-heartbeat for s8 on db1071
  • 15:22 herron: beginning cut over of codfw db servers (^db2.*) to codfw puppet 4 masters
  • 14:49 jynus@tin: Synchronized wmf-config/db-codfw.php: mariadb: Setup s8 replica set on codfw (duration: 00m 45s)
  • 14:27 moritzm: installing libxml-libxml-perl security updates
  • 14:21 jynus: starting database topology changes for s8 on codfw T177208
  • 14:11 urandom: bootstrapping cassandra, restbase2004-c.codfw.wmnet - T179422
  • 13:43 apergos: one more round of labstore1006 <-- ms1001 rsync catchup
  • 13:37 moritzm: installing imagemagick security updates
  • 12:39 marostegui: Stop MySQL on db1053 to clone db1097.s4 - T178359
  • 12:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T178359 (duration: 00m 45s)
  • 12:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Start adapting the config to move db1097 to s4 and s5 as multi-instance rc slave T178359 (duration: 00m 45s)
  • 12:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1101.s7 - T178359 (duration: 00m 45s)
  • 11:51 jynus: starting dropping incorrectly created database on s7 amwikimedia (not to be confused with production wiki s3 amwikimedia)
  • 11:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
  • 11:19 akosiaris: gnt-node evacuate -s -f ganeti1005. T181121
  • 11:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101.s7 - T178359 (duration: 00m 45s)
  • 10:54 akosiaris: gnt-node migrate -f ganeti1005. T181121
  • 10:51 marostegui: Drop index from ores_classification on s5 - T180045
  • 10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1051 and db1063 in vslow service for s5 to warm them up for the s8 split - T177208 (duration: 00m 45s)
  • 10:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1101 to s5 and s7 as recentchanges multi-instance slave - T178359 (duration: 00m 45s)
  • 10:04 moritzm: running "scap pull" on mw1191, it's depooled and marked as "inactive", but health checks are triggering db errors
  • 09:35 bblack: cr[12]-ulsfo - switch static fallback LVS routes from lvs400[12] to lvs400[56]
  • 09:27 bblack: lvs@ulsfo - done switching primaries (host MED config) - lvs400[56] now primary for text/upload traffic
  • 09:11 akosiaris@tin: Finished deploy [parsoid/deploy@b150764]: T180211 (duration: 05m 05s)
  • 09:08 bblack: puppet disabled on lvs400[1256] for switching primaries
  • 09:06 akosiaris@tin: Started deploy [parsoid/deploy@b150764]: T180211
  • 09:04 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp2017.codfw.wmnet
  • 09:00 bblack: lvs4005 - reboot to clear experimental stuff
  • 08:16 bblack: backend restart on cp4024 (upload@ulsfo) - mailbox lag
  • 07:56 marostegui: Drop index from ores_classification on s3 - T180045
  • 07:50 marostegui: Drop index from ores_classification on s6 - T180045
  • 07:48 marostegui: Drop index from ores_classification on s7 - T180045
  • 07:29 _joe_: stopping the additional workers for htmlCacheUpdate (commons and ruwiki), adding one additional runner for refreshLinks on ruwiki
  • 06:43 marostegui: Stop MySQL on db1063 and db1051 (which is going to be recloned) - T177208
  • 06:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1051 from s1 to s5 - T177208 (duration: 00m 53s)
  • 06:28 ejegg: updated payments-wiki from f871160 to 1ca91b1
  • 03:36 mutante: powercycled mw2251 which had gone down without further comment
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.8) (duration: 11m 34s)
  • 01:09 hoo: Cleaned out remaining T180934 related log blow up on snapshot1007 (dumpwikidatajson-wikidata-20171120-all-0.log)
  • 00:43 mutante: gerrit restarting service to apply config change
  • 00:40 mutante: gerrit - re-enabling puppet to apply logstash change on cobalt, gerrit restart incoming (T141324)
  • 00:39 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/#/c/392754/ (duration: 00m 47s)
  • 00:11 maxsem@tin: Synchronized php-1.31.0-wmf.8/extensions/InputBox/: https://gerrit.wikimedia.org/r/#/c/392745/ (duration: 00m 45s)
  • 00:10 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/392576/ (duration: 00m 46s)
  • 00:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=nescio.wikimedia.org
  • 00:04 bblack@neodymium: conftool action : set/pooled=no; selector: name=nescio.wikimedia.org
  • 00:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
  • 00:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=maerlant.wikimedia.org
  • 00:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=maerlant.wikimedia.org
  • 00:00 bblack@neodymium: conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org

2017-11-21

  • 23:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
  • 23:30 mholloway-shell@tin: Finished deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83 (duration: 04m 36s)
  • 23:26 mholloway-shell@tin: Started deploy [mobileapps/deploy@fc01242]: Update mobileapps to 52d6a83
  • 22:45 bblack@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org
  • 22:38 bblack@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
  • 22:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=hydrogen.wikimedia.org
  • 22:29 urandom: Bootstrapping Cassandra, restbase2004-b.codfw.wmnet (T179422)
  • 22:12 mutante: gerrit - temp disable puppet on cobalt (prod gerrit), test switching gerrit logging to logstash on gerrit2001 - gerrit:392079 gerrit:392083 T141324
  • 22:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
  • 22:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
  • 22:02 bblack: repooling acamar for recdns
  • 21:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 02s)
  • 21:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: take 2: gzip namespace and abstract dumps; remove last configfile existence checks
  • 20:45 bblack: recdns: puppet disabled on all, acamar depooled, careful deploys going on for anycast+recdns stuff
  • 20:42 ayounsi@neodymium: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
  • 20:26 ariel@tin: Finished deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks (duration: 00m 16s)
  • 20:26 ariel@tin: Started deploy [dumps/dumps@16f92d6]: gzip namespace and abstract dumps; remove last configfile existence checks
  • 20:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to 1.31.0-wmf.8
  • 20:00 thcipriani: finishing wmf.8 rollout, starting group2 to wmf.8
  • 19:21 addshore: SWAT done!
  • 19:20 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Add images.collection.cooperhewitt.org & *.dimu.org to wgCopyUploadsDomains. T180791 T180241 (duration: 00m 48s)
  • 19:11 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable AdvancedSearch on group0 PT2/2 (duration: 00m 49s)
  • 19:10 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable AdvancedSearch on group0 PT1/2 (duration: 00m 50s)
  • 18:40 mholloway-shell@tin: Finished deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d (duration: 05m 09s)
  • 18:35 mholloway-shell@tin: Started deploy [mobileapps/deploy@dd41387]: Update mobileapps to 9d1602d
  • 18:28 smalyshev@tin: Finished deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003 (duration: 01m 55s)
  • 18:26 smalyshev@tin: Started deploy [wdqs/wdqs@c69c739]: Restore categories vocabulary to V003
  • 18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003 (duration: 00m 11s)
  • 18:25 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
  • 16:45 marostegui: Compress s3 on db2085 - T178359
  • 16:35 papaul: powering down wtp2017 for disk replacement
  • 16:19 papaul: updating firmware on db2068
  • 15:31 _joe_: rolling restart of pybal on low-traffic in codfw, eqiad for the new depool thresholds for MW
  • 15:00 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication (duration: 00m 26s)
  • 15:00 ppchelko@tin: Started deploy [cpjobqueue/deploy@3e948de]: Revert: Temporarily disable deduplication
  • 14:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1096 (duration: 00m 48s)
  • 14:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication (duration: 00m 29s)
  • 14:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@aac3201]: Temporarily disable deduplication
  • 14:29 zeljkof: EU SWAT finished
  • 14:25 bblack: cr[12]-ulsfo: allow lvs400[567] as PyBal neighbors for BGP
  • 14:18 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayerOgvJs.js: SWAT: Disable wasm, use asm.js codec modules for Safari/Edge (T181022) (duration: 00m 49s)
  • 14:12 jdrewniak@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
  • 14:11 jdrewniak@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 50s)
  • 14:11 ppchelko@tin: Started restart [cpjobqueue/deploy@5341d94]: Restart to try increased maxSockets
  • 14:11 Pchelolo: restart cpjobqueue to try increasing maxSockets
  • 14:08 bblack: not-abnormally-quick reboot on lvs4005
  • 14:08 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916
  • 13:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting (duration: 00m 36s)
  • 13:44 ppchelko@tin: Started deploy [cpjobqueue/deploy@5341d94]: Enable GC metrics reporting
  • 13:39 _joe_: starting 2 manual runners for htmlcacheupdate on commons, 1 for htmlcacheupdate and 1 for refreshlinks on ruwiki, on terbium
  • 13:39 bblack: quick reboot on lvs4005
  • 12:02 kartik@tin: Finished deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987 (duration: 03m 24s)
  • 11:58 kartik@tin: Started deploy [cxserver/deploy@b87a27a]: Update cxserver to 4301987
  • 11:04 akosiaris: reboot ununpentium for serial tty change
  • 11:03 akosiaris: reboot oresrdb2001 for serial tty change
  • 11:02 akosiaris: reboot netmon1003 for serial tty change
  • 11:01 akosiaris: reboot install2002 for serial tty change
  • 10:57 akosiaris: reboot install1002 for serial tty change
  • 10:35 godog: bootstrap cassandra restbase2004-a - T179422
  • 10:11 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007 (duration: 00m 31s)
  • 10:10 ppchelko@tin: Started deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007
  • 09:39 elukey: upload prometheus-druid-exporter 0.4 to jessie/stretch-wikimedia
  • 09:28 marostegui: Shutdown db2068 for maintenance - T180927
  • 09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s)
  • 08:39 marostegui: Drop index on db1089 enwiki.ores_classification - T180045
  • 08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 with low weight - T178359 (duration: 00m 48s)
  • 08:27 marostegui: Reboot db1096 for kernel and MariaDB upgrade
  • 08:19 marostegui: Compress s5 on db1101 - T178359
  • 08:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1109 and db1110 - T180700 (duration: 00m 48s)
  • 07:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 and db1110 - T180700 (duration: 00m 49s)
  • 07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1109 and db1110 in s5 with small weight - T180700 (duration: 00m 48s)
  • 06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T179106 (duration: 00m 48s)
  • 06:47 marostegui: Remove index wb_terms_language from db1087 - https://phabricator.wikimedia.org/T179106
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T179106 (duration: 00m 48s)
  • 06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s)
  • 06:27 marostegui: Stop MySQL on db1096 to clone db1101.s5 - T178359
  • 06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T177208 (duration: 00m 49s)
  • 02:58 mutante: phabricator back up
  • 02:55 mutante: phab1001 (phabricator prod) reboot for kernel upgrade
  • 02:48 krinkle@tin: Synchronized docroot/noc: clean up (duration: 00m 49s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 42s)
  • 02:03 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003
  • 00:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: T180863 (duration: 00m 49s)

2017-11-20

  • 23:46 mutante: phab2001 - reboot for kernel upgrade
  • 23:35 legoktm@tin: Synchronized wmf-config/InitialiseSettings.php: emergency disable ORES on frwp/ruwp T181006 (duration: 00m 49s)
  • 23:35 awight: purge cache keys for ORES thresholds on frwiki and ruwiki
  • 23:25 awight@tin: Started deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006
  • 23:19 awight: aborted ORES rollback
  • 23:18 awight@tin: Started deploy [ores/deploy@95cd523]: Rollback ORES (take 2); 181006
  • 23:14 smalyshev@tin: Finished deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug (duration: 08m 11s)
  • 23:11 awight: purged memcache key 'ruwiki:ORES:threshold_statistics:goodfaith:1’, T181006
  • 23:06 smalyshev@tin: Started deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug
  • 22:55 awight@tin: Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s)
  • 22:55 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: no wmf.8 for group2. i hate my life
  • 22:55 awight: rolling back ORES to fix T181006
  • 22:54 eileen: update civicrm from 272580a to a16e566
  • 22:54 awight@tin: Started deploy [ores/deploy@5084251]: Rollback ORES; T179711
  • 22:50 Krinkle: MW HTTP 500 spike tracked as https://phabricator.wikimedia.org/T181006
  • 22:49 Krinkle: Sharp rise in HTTP 500 errors as of 22:05 (45 minutes ago)
  • 22:29 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: fix cache key namespace (duration: 00m 49s)
  • 22:27 awight@tin: Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s)
  • 22:26 aaron@tin: Synchronized php-1.31.0-wmf.7/extensions/Collection: cache key name fix (duration: 00m 49s)
  • 22:11 aaron@tin: Synchronized php-1.31.0-wmf.8/includes/libs/objectcache/WANObjectCache.php: 7e74b49: namespace WAN cache variant keys (duration: 00m 48s)
  • 22:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8
  • 21:52 bblack: ns2 back on eeden
  • 21:52 aaron@tin: Synchronized php-1.31.0-wmf.8/extensions/Collection: 3baebf4a: cache key name fix (duration: 00m 51s)
  • 21:44 bblack: rebooting eeden
  • 21:37 awight@tin: Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711
  • 21:33 bblack: routing ns2.wikimedia.org to radon for eeden reboot
  • 21:19 bblack: routing ns0.wikimedia.org back to radon post-reboot
  • 21:13 bblack: rebooting radon
  • 21:07 bblack: re-routing ns0.wikimedia.org traffic to baham for radon reboot
  • 20:17 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 19m 41s)
  • 20:16 bblack: ns1 dns traffic back to normal on baham
  • 20:16 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Translate extension in amwikimedia (T180879)" (duration: 00m 48s)
  • 20:11 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Translate extension in amwikimedia (T180879) (duration: 00m 49s)
  • 20:11 hashar: CI docker jobs were all broken due to a mistake. Should be back now. T177684
  • 20:07 ladsgroup@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: RCFilters: Only apply excluded label to namespace items (T180863) (duration: 00m 49s)
  • 20:06 bblack: rebooting baham
  • 19:58 bblack: re-routing ns1.wikimedia.org traffic to radon for baham reboot
  • 19:52 chasemp: updating phab mail handler
  • 19:51 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 00m 49s)
  • 19:47 ottomata: restarted coal with fixes for eventcapsule changes in T179625
  • 19:42 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 49s)
  • 19:40 ladsgroup@tin: Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 48s)
  • 19:36 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow admins to remove users from MP3 uploaders user group (T180002) (duration: 00m 49s)
  • 19:31 ladsgroup@tin: Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 51s)
  • 19:30 ladsgroup@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 49s)
  • 19:22 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Switch submit button from 'save' to 'publish' on dewiki (duration: 00m 50s)
  • 18:51 chasemp: disable puppet across cloud things to rollout https://gerrit.wikimedia.org/r/#/c/392168/ slowly
  • 18:25 smalyshev@tin: Finished deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update (duration: 01m 42s)
  • 18:23 smalyshev@tin: Started deploy [wdqs/wdqs@2e39b69]: Blazegraph and GUI update
  • 17:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T177208 (duration: 00m 49s)
  • 17:19 bd808: Sending Toolforge survey emails from silver for T177126
  • 17:00 moritzm: uploaded git 2.11-3+deb9u2+bpo8+wmf1 for component/git to apt.wikimedia.org/jessie-wikimedia
  • 16:43 marostegui: Reboot db1082 for kernel upgrade and MariaDB upgrade to 10.0.33
  • 16:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev (duration: 03m 33s)
  • 16:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
  • 16:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)
  • 16:36 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Revert: Temporary set consumer_batch_size to 50, forgot -f, checkout prev rev
  • 16:32 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f (duration: 00m 28s)
  • 16:32 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50, forgot -f
  • 16:24 marostegui: Add db1109 and db1110 to tendril - T180700
  • 16:06 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50 (duration: 00m 17s)
  • 16:06 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Revert: Temporary set consumer_batch_size to 50
  • 15:59 urandom: Use lz4 compression instead of deflate (T180804)
  • 15:45 jynus: shutting down db2068 for maintenance after depool T180927
  • 15:42 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2068 (duration: 00m 49s)
  • 15:34 chasemp: disable puppet for a merge across cloud things
  • 15:31 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50 (duration: 00m 30s)
  • 15:30 ppchelko@tin: Started deploy [cpjobqueue/deploy@f0610d3]: Temporary set consumer_batch_size to 50
  • 15:03 herron: pointing codfw mw servers at codfw puppet 4 masters via puppetmaster2001
  • 14:39 zeljkof: EU SWAT finished
  • 14:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Single edit tab in Catalan Wikipedia (T180660) (duration: 00m 49s)
  • 14:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable wgNamespacesWithSubpages for hiwikiversity (T180913) (duration: 00m 48s)
  • 14:22 jynus: enable semi-sync replication on s5
  • 14:14 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180805: Revert [cirrus] disable token count router (duration: 00m 49s)
  • 14:05 elukey: upload prometheus-druid-exporter 0.3 to jessie-wikimedia
  • 13:53 Pchelolo: disable puppet on scb100x and stop cpjobqueue to accumulate some backlog
  • 13:39 ppchelko@tin: Finished deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation (duration: 00m 28s)
  • 13:39 ppchelko@tin: Started deploy [cpjobqueue/deploy@174420f]: Optimise committed offset calculation
  • 13:30 elukey: upload prometheus-druid-exporter 0.3 to stretch-wikimedia
  • 12:40 marostegui: Optimize db1063 wikidatawiki.wb_terms
  • 12:17 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+icu57 to apt.wikimedia.org (jessie-wikimedia/component/icu57) (HHVM build linked against a co-installable backport of icu57)
  • 11:46 moritzm: rebooting tungsten for update to 4.9.51
  • 11:41 moritzm: rebooting etherpad1001 (etherpad.wikimedia.org) for update to 4.9.51
  • 11:36 moritzm: rebooting pybal-test for update to 4.9.51
  • 11:27 moritzm: rebooting restbase-test cluster for update to 4.9.51
  • 11:23 ppchelko@tin: Finished deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing (duration: 00m 30s)
  • 11:22 ppchelko@tin: Started deploy [cpjobqueue/deploy@bdcef23]: Various performance improvements in committing
  • 11:13 moritzm: rebooting ruthenium for update to 4.9.51
  • 10:57 godog: reimage restbase2004 - T179422
  • 10:51 moritzm: rebooting cerium/praseodymium/xenon for update to 4.9.51
  • 10:42 moritzm: rebooting mwlog1001 for update to 4.9.51
  • 10:39 moritzm: rebooting mwlog2001 for update to 4.9.51
  • 10:23 hoo: Manually re-started the Wikidata entity JSON dump on snapshot1007 (T180934)
  • 10:19 dcausse: elastic/cirrus: reindexing english group0 and group1 wikis: T179945
  • 10:08 hashar: contint1001: sudo systemctl start jenkins
  • 10:05 moritzm: rebooting contint1001 for update to 4.9.51
  • 09:22 moritzm: rebooting hafnium for update to 4.9.51
  • 08:46 marostegui: Run mydumper for db1047.staging - T156844
  • 07:58 moritzm: installing procmail security updates
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T178359 (duration: 00m 49s)
  • 06:41 marostegui: Stop MySQL on db1082 to clone db1109 and db1110 - T180700
  • 06:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T177208 (duration: 00m 48s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original weights for db1100 and db1071 - T180917 (duration: 00m 49s)
  • 06:15 marostegui: Reboot db2068 - T180927
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 06m 26s)

2017-11-19

  • 23:20 Jamesofur: removed 2FA for Ask21 T180889
  • 20:28 krinkle@tin: Synchronized docroot/noc/index.html: noc: Link to Grafana (duration: 00m 49s)
  • 19:50 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
  • 07:09 mobrovac@tin: Started restart [zotero/translators@a0c41c3]: Zotero eating up memory
  • 07:01 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: electron stuck - T174916

2017-11-18

  • 20:06 gehel: upgrade of elasticsearch eqiad complete - T178411
  • 18:03 reedy@tin: Synchronized multiversion/vendor/: bump (duration: 01m 14s)
  • 15:53 marostegui: Compress s5 on db2085 - T178359
  • 14:48 marostegui: Compress s3 on db2092 - T178359
  • 00:30 urandom: Bootstrapping restbase2006-b (T179422)

2017-11-17

  • 20:35 mutante: netmon1002 - rsync smokeping data back from local backup to show measurements made from eqiad as requested on T180812
  • 20:05 mutante: running puppet on all cache misc to switch smokeping web to eqiad
  • 20:02 mutante: T180812 copying smokeping data from 2001 to 1002 - netmon1002: /usr/bin/rsync -avp rsync://netmon2001.wikimedia.org/var-lib-smokeping /var/lib/smokeping/ | switching backend from codfw to eqiad
  • 19:37 mutante: T180812 - @netmon1002:# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/@netmon2001:/# rsync -avp /var/lib/smokeping/ /root/backup/netmon2001/201711717/var/lib/smokeping/
  • 19:36 mutante: @netmon1002:/var/lib/smokeping# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/
  • 18:37 anomie@tin: Synchronized wmf-config/InitialiseSettings.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 48s)
  • 18:36 anomie@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Setting wgCommentTableSchemaMigrationStage = MIGRATION_WRITE_BOTH on Beta Cluster, no prod change (duration: 00m 49s)
  • 17:43 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 49s)
  • 17:41 demon@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase/client/WikibaseClient.php: fix client dependencies (duration: 00m 50s)
  • 17:21 bblack: enabling new normalization code for all upload@esams (done)
  • 17:19 bblack: enabling new normalization code for all upload@eqiad
  • 17:14 marostegui: Revert schema change on dbstore1001 - T180714
  • 17:12 marostegui: Move dbstore1001 under db1070 - T180714
  • 17:12 bblack: enabling new normalization code for all upload@codfw
  • 16:52 bblack: enabling new normalization code for all upload@ulsfo
  • 16:24 bblack: disabling puppet on all cp* (testing encoding patch)
  • 16:19 moritzm: uploaded boost 1.55.0+dfsg-3+wmf2+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
  • 16:04 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T180795 [cirrus] disable token count router (duration: 00m 49s)
  • 15:43 chasemp: labservices1001:~# mv /var/zones/tools.eqiad.wmflabs /home/rush T180797
  • 15:26 urandom: Starting restbase2006-c w/ -Dcassandra.replace_address=10.192.48.51 (T179422)
  • 14:06 jynus: deploy master events to db1070
  • 14:04 chasemp: labstore1003 service nfs-kernel-server restart && service rsync start
  • 13:55 moritzm: uploaded boost 1.55.0+dfsg-3+wmf1+icu57 to apt.wikimedia.org for jessie-wikimedia/component/icu57 (needed for HHVM build linked against ICU 57)
  • 12:15 moritzm: installing openssl updates on poolcounters
  • 12:07 moritzm: installing openssl updates on graphite* hosts
  • 11:46 moritzm: installing openssl updates on etcd* hosts
  • 11:43 moritzm: installing openssl updates on dbproxy hosts
  • 11:33 moritzm: installing openssl updates on puppetmasters
  • 11:01 akosiaris: create webperf1001, webperf2001 in ganeti T179036
  • 10:50 akosiaris@tin: Synchronized wmf-config/db-eqiad.php: (no justification provided) (duration: 00m 49s)
  • 10:49 akosiaris: sync wmf-config/db-eqiad.php for T180724
  • 10:48 akosiaris@tin: scap aborted: (no justification provided) (duration: 00m 02s)
  • 10:48 akosiaris@tin: Started scap: (no justification provided)
  • 10:47 akosiaris: pool mw2251 T180724
  • 10:47 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet
  • 10:01 moritzm: rebooting darmstadtium (docker registry) for update to 4.9.51
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore original traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 48s)
  • 09:54 moritzm: rebooting labnodepool* for update to 4.9.51
  • 09:38 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more (duration: 00m 29s)
  • 09:38 ppchelko@tin: Started deploy [cpjobqueue/deploy@2141162]: Bump concurrency even more
  • 09:20 godog: start restbase2006-c instead, restbase2006-b failed and -c shows as "down" - T179422
  • 09:13 jynus: changing master of db1071 (63 -> 70)
  • 09:10 gehel: cleanup leftover titlesuggest indices on elasticsearch eqiad (jawiki, frwiki, ptwiki)
  • 09:05 ppchelko@tin: Finished deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog (duration: 00m 35s)
  • 09:04 ppchelko@tin: Started deploy [cpjobqueue/deploy@cbd25d3]: Bump overall concurrency to get rid of RecordLintJob backlog
  • 08:54 godog: bootstrap restbase2006-b - T179422
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for the s5 eqiad hosts that were out for hours due to schema change revert after s5 master crash (duration: 00m 50s)
  • 08:04 elukey: reboot stat100[456] for kernel updates
  • 07:51 marostegui: Revert schema change on dbstore1002 - T180714
  • 07:50 marostegui: Move dbstore1002 under db1070 - T180714
  • 07:33 marostegui: Enable GTID on all the eqiad up-to-date hosts, only pending db1071 - T180714
  • 07:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool s5 eqiad hosts after reverting schema change - T180714 (duration: 00m 50s)
  • 06:34 marostegui: Revert schema changes on s5 codfw master with replication enabled, lag will be generated on codfw s5 - T180714
  • 04:43 krinkle@tin: Synchronized wmf-config/CommonSettings-labs.php: no-op for labs (duration: 00m 51s)
  • 01:32 ejegg: re-enabled donation queue consumer
  • 01:28 ejegg: updated CiviCRM from 8454e06 to 272580a
  • 01:07 ejegg: updated CiviCRM from 0b8ceea to 8454e06
  • food: disabled donations queue consumer for thank you subject update
  • 00:41 krinkle@tin: Synchronized wmf-config/PhpAutoPrepend.php: I60cce0 (duration: 00m 48s)
  • 00:40 krinkle@tin: Synchronized wmf-config/StartProfiler.php: I60cce0 (duration: 00m 49s)
  • 00:37 krinkle@tin: Synchronized wmf-config/profiler.php: I60cce0 (duration: 00m 48s)
  • 00:23 urandom: Decommissioning Cassandra, restbase1014-c.eqiad.wmnet (T179422)
  • 00:22 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/390131/3 (duration: 00m 49s)

2017-11-16

  • 23:50 mutante: wezen - systemctl restart rsyslog
  • 21:52 mepps: updated civicrm from 22f6532 to 0b8ceea
  • 21:05 mepps: updated payments-wiki from 210cb37 to f871160
  • 20:51 apergos: one more catchup rsync from ms1001 to labstore1006 kicking off
  • 20:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - third try (duration: 00m 49s)
  • 20:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - second try (duration: 00m 49s)
  • 20:10 demon@tin: Unlocked for deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (duration: 06m 49s)
  • 20:03 demon@tin: Locking from deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (planned duration: 360m 00s)
  • 20:02 twentyafterfour: deploy a9613b4 to hotfix T180706
  • 19:56 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (duration: 50m 28s)
  • 19:54 ayounsi@tin: Finished deploy [netbox/deploy@19f4f65]: (no justification provided) (duration: 00m 04s)
  • 19:54 ayounsi@tin: Started deploy [netbox/deploy@19f4f65]: (no justification provided)
  • 19:49 akosiaris: schedule extra downtime for s5 slaves
  • 19:46 urandom: Bootstapping Cassandra restbase2006-a (T179422)
  • 19:13 jynus: reset slave all on db1070 @ db1063-bin.001382:234464548
  • 19:05 demon@tin: Locking from deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (planned duration: 360m 00s)
  • 19:05 demon@tin: Unlocked for deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (duration: 01m 22s)
  • 19:03 demon@tin: Locking from deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (planned duration: 60m 00s)
  • 18:48 marostegui: dewikipedia and wikidata currently back to writable
  • 18:44 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers (duration: 00m 48s)
  • 18:41 mutante: dewikipedia and wikidata currently back to readonly-mode while wikidata is being worked on
  • 18:41 marostegui: Set s5 master read_only
  • 18:25 akosiaris: set mw2251 to inactive. T180724
  • 18:24 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2251.codfw.wmnet
  • 17:57 gehel: silencing wdqs
  • 17:55 jynus@tin: Synchronized wmf-config/db-eqiad.php: Failover db1063 to db1070 (duration: 00m 46s)
  • 17:44 XioNoX: merging netbox CR
  • 17:41 bblack: disable port ge-5/0/39 on asw-c-eqiad (db1063)
  • 17:40 urandom: Decommissioning Cassandra, restbase1014-b.eqiad.wmnet (T179422)
  • 17:36 jynus: stopping slave on db1070
  • 17:28 urandom: Converting 'enwiki parsoid' to size-tiered compaction (T179422)
  • 17:19 demon@tin: Finished scap: consistency (duration: 25m 12s)
  • 17:18 bblack: Temporary GeoDNS routing changes (eqsin traffic simulation using ulsfo) - https://gerrit.wikimedia.org/r/#/c/391357/ - expecting ~24h, West Asia latencies will probably increase, spike in cache misses, etc...
  • 17:18 urandom: Converting 'wikipedia parsoid' to size-tiered compaction (T179422)
  • 17:16 godog: reimage restbase2006 - T179422
  • 17:15 urandom: Converting 'commons mobile' to size-tiered compaction (T179422)
  • 17:05 urandom: Converting 'others mobile' to size-tiered compaction (T179422)
  • 16:54 demon@tin: Started scap: consistency
  • 16:54 godog: reimage restbase2006 - T179422
  • 16:48 jynus: stop and restart db1071 for upgrade and reconfiguration
  • 16:48 herron: beginning gradual cutover of codfw mw systems to puppet 4 master puppetmaster2001
  • 16:48 godog: upgrade hpsa firmware to 6.06 on restbase2006 - T141756
  • 16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1034 and db1079 original weight (duration: 00m 48s)
  • 16:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: mariadb: Depool db1071, pool db1104 as api (duration: 00m 49s)
  • 16:21 moritzm: restarting apache on labs puppet masters to pick up openssl updates
  • 16:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
  • 15:47 moritzm: rebooting ores1* for update to 4.9.51
  • 15:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 49s)
  • 15:33 moritzm: rebooting seaborgium (slapd) for update to 4.9.51
  • 15:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
  • 15:16 milimetric@tin: Finished deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset (duration: 14m 29s)
  • 15:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s)
  • 15:10 godog: upgrade prometheus-redis-exporter to 0.13-1 - T148637
  • 15:06 moritzm: installing postgres security updates on labsdb1006/1007
  • 15:02 milimetric@tin: Started deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset
  • 15:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T177208 (duration: 00m 48s)
  • 15:00 herron: beginning cut over of codfw canary appservers to puppet 4 master puppetmaster2001
  • 14:55 moritzm: rebooting serpens (slapd) for update to 4.9.51
  • 14:50 elukey: updating puppet compiler's facts (following https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Puppet3-diffs#FAQ)
  • 14:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
  • 14:26 dcausse: EU swat done
  • 14:23 moritzm: rebooting kubetcd* to 4.9.51
  • 14:18 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: T177913: [cirrus] Add overridden iw prefix for svwiki (2/2) (duration: 00m 48s)
  • 14:16 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T177913: [cirrus] Add overridden iw prefix for svwiki (1/2) (duration: 00m 50s)
  • 14:02 gehel: starting upgrade of elasticsearch eqiad - T178411
  • 13:46 moritzm: rebooting bohrium (piwik host) for update to 4.9.51
  • 13:33 moritzm: installing openssl updates on es* hosts
  • 13:24 moritzm: rebooting dubnium/pollux (openldap corp mirror) for update to 4.9.51
  • 13:12 moritzm: installing openssl updates on pc* hosts
  • 13:07 elukey: restart aqs on aqs100[5-9] to apply localQuorum (https://gerrit.wikimedia.org/r/391765) - T164348
  • 13:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 49s)
  • 12:54 moritzm: rebooting radium (tor relay) for update to 4.9.51
  • 12:34 moritzm: installing openssl updates on memcached/redis clusters
  • 12:27 hoo: Updated operations/dumps/dcat (ea4e75..7734e04) on snapshot1007
  • 12:09 moritzm: rebooting dbmonitor* hosts for update to 4.9.51
  • 12:03 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikidata/extensions/Constraints: T180665 Manually applied: Fix SparqlHelper::getCacheMaxAge() (duration: 00m 52s)
  • 11:58 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180665 Fix SparqlHelper::getCacheMaxAge() (duration: 00m 53s)
  • 11:42 moritzm: installing openssl updates on conf* clusters
  • 11:37 addshore@tin: Synchronized wmf-config/Wikibase-production.php: testwikidata only, T180665, WBQC configuration for testwikidatawiki (duration: 00m 49s)
  • 11:32 addshore: addshore@terbium:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintStatements.php --wiki=testwikidatawiki > importConstraintStatements.log
  • 11:21 godog: upgrade prometheus to 1.8.1+ds+k8s-1 in ulsfo/esams/eqiad - T177395
  • 11:09 moritzm: rebooting debug proxies (hassium/hassaleh) for update to 4.9.51
  • 11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1034 weight - T178359 (duration: 00m 48s)
  • 11:01 jynus: shutting down labsdb1010 to clone to labsdb1009
  • 10:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359 (duration: 00m 48s)
  • 10:26 addshore: Kill the build deploy slot done!
  • 10:24 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/Wikibase: T180634 Bring Wikibase up to date with .8 branch of Wikidata build extension (duration: 01m 46s)
  • 10:15 addshore@tin: Synchronized php-1.31.0-wmf.8/extensions/WikibaseQualityConstraints: T180634 Bring WikibaseQualityConstraints up to date with .8 branch of Wikidata build extension (duration: 00m 53s)
  • 09:59 moritzm: rebooting prometheus servers in eqiad for update to 4.9.51
  • 09:44 elukey: restart aqs on aqs1004 to apply localQuorum (https://gerrit.wikimedia.org/r/391765) - T164348
  • 09:38 moritzm: rebooting prometheus servers in codfw for update to 4.9.51
  • 09:30 moritzm: uploaded icu57.1-6+wmf2 to jessie-wikimedia/component/icu57
  • 09:23 godog: bootstrap restbase2002-c - T179422
  • 09:19 godog: upgrade grafana to 4.6.1 on https://grafana.wikimedia.org/ - T180428
  • 08:19 marostegui: Deploy schema change on db1092 - T174569
  • 08:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174569 (duration: 00m 49s)
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 49s)
  • 07:40 marostegui: Stop MySQL on db1034 to copy its content to db1101.s7 - T178359
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359 (duration: 00m 49s)
  • 07:17 ppchelko@tin: Finished deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017 (duration: 00m 14s)
  • 07:17 ppchelko@tin: Started deploy [eventlogging/eventbus@872cfb3]: Revert gerrit 302372 due to AssertionError T180017
  • 06:54 marostegui: Deploy alter table on db1096 - T174569
  • 06:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174569 (duration: 00m 48s)
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 and db1071 - T174569 (duration: 00m 49s)
  • 06:15 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 28s)
  • 06:14 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 54s)
  • 02:09 twentyafterfour: phabricator database migrations complete, service is back online
  • 01:28 twentyafterfour: Phabricator will be offline for a couple of minutes while I apply database migrations.
  • 01:22 twentyafterfour: updating phabricator (belatedly)
  • 01:05 demon@tin: Synchronized scap/plugins/prep.py: no-op, co-master sync (duration: 00m 55s)
  • 00:59 tzatziki: Removing 2FA from AuburnPilot (T180654)
  • 00:59 tzatziki: Removing 2FA from Twotwo2019 (T180438)
  • 00:42 eileen: update civicrm from b99a9cf to 22f6532

2017-11-15

  • 22:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidatawiki back to wmf.7
  • 22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.8
  • 22:11 urandom: Setting keyspaces erroneously configured for leveled compaction, to use size-tiered (T180568)
  • 22:05 madhuvishy: Kicking off re-enabling puppet and puppet runs across Cloud VPS instances
  • 21:33 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 21:21 madhuvishy: Disabling puppet across cloud VPS through cumin on labpuppetmaster1001
  • 21:01 ottomata: restarting kafka-jumbo brokers to update to Kafka 0.11.0.1
  • 20:52 urandom: Restarting restbase2002-a.codfw.wmnet (T180568)
  • 20:40 mutante: restarting gerrit to enable 'large file support'-plugin gerrit:391635
  • 20:38 addshore@tin: Finished scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539 (duration: 21m 48s)
  • 20:17 addshore@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (again) T180539
  • 20:11 otto@tin: Finished deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/ (duration: 00m 20s)
  • 20:11 otto@tin: Started deploy [eventlogging/eventbus@872cfb3]: deploying kafka-futures change to kafka1001 only, will apply async with https://gerrit.wikimedia.org/r/#/c/391634/
  • 19:53 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add BP and WP aliases for project namespace on mwlwiki (T180052) (duration: 00m 48s)
  • 19:48 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on mwlwiki (T180052) (duration: 00m 48s)
  • 19:41 demon@tin: Finished deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9 (duration: 00m 08s)
  • 19:41 demon@tin: Started deploy [gerrit/gerrit@bce982f]: adding lfs plugin @ 2.13.9
  • 19:40 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Disable EventLogging for Popups (T178500) (duration: 00m 49s)
  • 19:37 gehel: upgrade of elasticsearch codfw completed, cluster still recovering - T178411
  • 19:34 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Minerva download icon on all wikis (T179914) (duration: 00m 48s)
  • 19:30 chasemp: labstore1004:~# service nfs-kernel-server restart
  • 19:19 catrope@tin: Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/mw.rcfilters.UriProcessor.js: T180577 (duration: 00m 49s)
  • 19:16 catrope@tin: Synchronized php-1.31.0-wmf.7/skins/MinervaNeue/resources/skins.minerva.scripts/init.js: Limit download button to Google Chrome (T179529, T179914) (duration: 00m 49s)
  • 19:14 catrope@tin: Synchronized wmf-config/flaggedrevs.php: Enable RCFilters on all remaining wikis (T177445) (duration: 00m 49s)
  • 19:01 urandom: Restarting Cassandra instances on restbase2001.codfw.wmnet (T180568)
  • 18:38 Jamesofur: removed 2FA from User:Einsbor after verification for votewiki, stewardwiki and SUL
  • 18:20 moritzm: rebooting auth* servers for update to 4.9.51
  • 18:20 moritzm: rebooting auth
  • 17:57 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: removing support for non-wikipedias (duration: 00m 49s)
  • 16:03 addshore@tin: Finished scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539) (duration: 23m 48s)
  • 15:56 oblivian@tin: Finished deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images (duration: 00m 18s)
  • 15:56 oblivian@tin: Started deploy [docker-pkg/deploy@9b319d2]: Adding do extension to jinja2, needed for contint images
  • 15:55 ema: restart pybal on lvs[12]00[36] for config change https://gerrit.wikimedia.org/r/#/c/389964/
  • 15:43 jynus: shutting down labsdb1009 for maintenance T179244
  • 15:40 addshore@tin: Started scap: Revert partial scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539)
  • 15:29 moritzm: installing perl security updates on trusty (Debian hosts fixed two months ago)
  • 15:14 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539) (duration: 03m 48s)
  • 15:13 moritzm: removing unused kernels from prometheus*
  • 15:11 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch, second try (T180539)
  • 15:05 moritzm: rebooting kubestagetcd* for update to 4.9.51
  • 14:57 ladsgroup@tin: scap aborted: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539) (duration: 00m 31s)
  • 14:57 ladsgroup@tin: Started scap: Update extensions/Wikidata to new wmf/1.31.0-wmf.8 branch (T180539)
  • 14:42 Amir1: deployed ORES change for wikidata (gerrit:391197, phab:T180450)
  • 14:41 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 49s)
  • 14:37 mobrovac: restbase creating Cassandra 3 revision tables on restbase1009 - T179421
  • 14:30 zfilipin@tin: Synchronized wmf-config/: SWAT: Remove wgContentTranslationEnableSuggestions (duration: 00m 51s)
  • 14:27 ema: cache_upload: upgrade varnish to 4.1.8-1wm2
  • 14:22 zfilipin@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT: Beta: Explicitly set cookieDomain for ContentTranslationSiteTemplates (T149879) (duration: 00m 49s)
  • 14:18 ema: repool cp4024 T174891
  • 14:17 mutante: wtp2017 - systemctl start ferm (ferm wasnt running due to failed DNS lookup for prometheus2003 sometime in the past)
  • 14:12 zfilipin@tin: Synchronized php-1.31.0-wmf.8/extensions/EventBus/EventBus.php: SWAT: Increase request timeout to match kafka produce timeout (T180017) (duration: 00m 50s)
  • 14:11 godog: upgrade hpsa firmware to 6.06 on restbase2004 - T180562 T141756
  • 14:10 ema: upgrade varnish to 4.1.8-1wm2 on cp4024 (cache_upload, depooled)
  • 14:04 jynus: reload haproxy on dbproxy1010
  • 14:00 moritzm: installing openssl updates on kafka and hadoop clusters
  • 13:34 ema: cache_text: upgrade varnish to 4.1.8-1wm2
  • 13:31 ppchelko@tin: Started restart [changeprop/deploy@065a06e]: Restart to rebalance all rules T179684
  • 13:14 moritzm: rebooting video scalers in eqiad for update to 4.9.51 (and to pick up openssl update)
  • 13:07 ema: upgrade varnish to 4.1.8-1wm2 on cp3030 (cache_text)
  • 12:41 elukey: re-enable eventlogging after maintenance
  • 12:24 jynus: deploying dns change for m4-master
  • 12:19 ema: cache_misc: upgrade varnish to 5.1.3-1wm3
  • 12:09 elukey: executed sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 on all jobrunners
  • 12:01 ema: upgrade varnish to 5.1.3-1wm3 on cp3007 (cache_misc)
  • 11:58 ema: varnish 4.1.8-1wm2 uploaded to apt.w.o (main)
  • 11:50 ema: varnish 5.1.3-1wm3 uploaded to apt.w.o (experimental)
  • 11:45 jynus: restart haproxy on dbproxy1009
  • 11:32 marostegui: Stop MySQL on db1046
  • 10:58 moritzm: rebooting job runners in eqiad for update to 4.9.51 (and to pick up openssl update)
  • 10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106, optimizing wikidatawiki.wb_terms (duration: 00m 49s)
  • 10:09 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786 (duration: 04m 51s)
  • 10:04 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 4 T179786
  • 10:04 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786 (duration: 04m 44s)
  • 09:59 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 3, force T179786
  • 09:59 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786 (duration: 03m 32s)
  • 09:58 ppchelko@tin: (no justification provided)
  • 09:55 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x attempt 2 T179786
  • 09:55 ppchelko@tin: Finished deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786 (duration: 02m 44s)
  • 09:53 godog: reboot restbase2004 - T180562
  • 09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
  • 09:52 mobrovac@tin: Started restart [restbase/deploy@c76a665]: Pick up the new seeds definition - T179422
  • 09:52 ppchelko@tin: Started deploy [trending-edits/deploy@a0e1fe3]: Update node-rdkafka to v1.x T179786
  • 09:32 godog: restart cassandra on restbase2002-b - T180568
  • 09:30 moritzm: updating openssl on database hosts
  • 09:19 godog: restart cassandra on restbase2001-c - T180568
  • 09:15 marostegui: Stop mysql on db1046 to transfer its content to db1107 - T177405
  • 09:08 elukey: stop eventlogging on eventlog1001, eventlogging replication on db1108/db1047/dbstore1002 as preparation steps to migrate the log db from db1046 to db1107
  • 09:02 jynus: rebooting labsdb1010 for kernel upgrade
  • 08:51 elukey: reboot thorium (hosting all analytics websites) for kernel updates
  • 08:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - going to convert it to mult-instance T178359 (duration: 00m 48s)
  • 07:49 marostegui: Deploy schema change on db1071 and db1099 (s5) - T174569
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 and db1071 - T174569 (duration: 00m 49s)
  • 07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 and db1100 - T174569 (duration: 00m 49s)
  • 05:33 subbu: subbu@terbium (followup to the linter-reparse.py script log entry) it is safe to kill -9 the script on terbium anytime if there are any problems because of it
  • 05:27 subbu: subbu@terbium running linter-reparse.py script to initialize baseline linter categories for all wikis (12 hours in so far .. expected to run for ~2 weeks). hits parsoid eqiad cluster.
  • 04:06 demon@tin: Synchronized docroot/search.wikimedia.org/robots.txt: go away robots / kill some 404s (duration: 00m 50s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 57s)
  • 00:59 ejegg: updated payments-wiki from d150287 to 210cb37
  • 00:53 Reedy: going to restart zuul as it's got backed up
  • 00:33 maxsem@tin: Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/391220/ (duration: 00m 49s)
  • 00:18 ejegg: updated SmashPig payments listener from 5262d53 to 45aa626
  • 00:12 ejegg: updated CiviCRM from f571c67 to b99a9cf

2017-11-14

  • 23:29 mutante: releases1001 - chmod g+w /srv/org/wikimedia/releases/mediawiki/1.*
  • 23:01 urandom: Decommissioning Cassandra, restbase1014-a.eqiad.wmnet (T179422)
  • 22:57 demon@tin: Synchronized w/: Removing mobilelanding from global w/, few sites actually need it (duration: 00m 49s)
  • 22:55 demon@tin: Synchronized docroot/m.wikipedia.org/w/mobilelanding.php: symlink -> real file (duration: 00m 50s)
  • 21:42 no_justification: gerrit: restarted services on master/cobalt, things will flap for a second
  • 21:38 mutante: gerrit2001 - restarted gerrit
  • 21:17 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master (duration: 02m 11s)
  • 21:15 ebernhardson@tin: Started deploy [search/mjolnir/deploy@494e0c6]: redeploy mjolnir submodule bump to master
  • 21:03 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master (duration: 02m 05s)
  • 21:01 ebernhardson@tin: Started deploy [search/mjolnir/deploy@77310bc]: update mjolnir to master
  • 20:37 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.8
  • 20:15 bblack: reboot cp4024
  • 19:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment (duration: 02m 12s)
  • 19:57 urandom: Bootstrapping restbase2002-b.codfw.wmnet (T179422)
  • 19:56 ebernhardson@tin: Started deploy [search/mjolnir/deploy@b20f0da]: test mjolnir deployment
  • 19:47 demon@tin: Finished scap: bootstrap wmf.8 (duration: 44m 37s)
  • 19:32 chasemp: clean up instances in error state in testlabs project
  • 19:10 jynus: restart db2034 for mariadb upgrade
  • 19:09 chasemp: for i in `OS_TENANT_NAME=testlabs openstack server list | grep stress | awk '{print $2}'`; do echo $i; OS_TENANT_NAME=testlabs openstack server delete $i; sleep 30; done T171473
  • 19:02 demon@tin: Started scap: bootstrap wmf.8
  • 18:32 arlolra: Updated Parsoid to e71937d0 (T178253)
  • 18:31 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862 (duration: 29m 12s)
  • 18:28 moritzm: upgraded nginx on notebook* to 1.13.6
  • 18:26 madhuvishy: Upgraded notebook1001 and 1002 to kernel version 4.9.51-1~bpo8+1
  • 18:22 arlolra@tin: Finished deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0 (duration: 09m 13s)
  • 18:21 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment (duration: 01m 13s)
  • 18:19 ebernhardson@tin: Started deploy [search/mjolnir/deploy@e6905f4]: test mjolnir deployment
  • 18:13 arlolra@tin: Started deploy [parsoid/deploy@b150764]: Updating Parsoid to e71937d0
  • 18:08 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided) (duration: 04m 28s)
  • 18:07 moritzm: rebooting notebook* hosts for update to 4.9.51
  • 18:04 ebernhardson@tin: Started deploy [search/mjolnir/deploy@cd6ddda]: (no justification provided)
  • 18:02 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Redeploying: Update mobileapps to c002862
  • 17:58 ebernhardson@tin: Finished deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided) (duration: 00m 20s)
  • 17:57 ebernhardson@tin: Started deploy [search/mjolnir/deploy@ceb5c2f]: (no justification provided)
  • 17:56 ebernhardson@tin: Finished deploy [search/MjoLniR/deploy@607adfb]: (no justification provided) (duration: 00m 14s)
  • 17:56 ebernhardson@tin: Started deploy [search/MjoLniR/deploy@607adfb]: (no justification provided)
  • 17:44 herron: restarted puppetdb on nitrogen and nihal to pick up jre updates
  • 17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 16s)
  • 17:42 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
  • 17:27 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 (duration: 07m 58s)
  • 17:12 demon@tin: Synchronized wmf-config/CommonSettings.php: Beta-only, no-op (duration: 00m 43s)
  • 17:11 demon@tin: Synchronized wmf-config/extension-list-labs: No-op (duration: 00m 44s)
  • 17:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.6 [keeping static files] (duration: 02m 48s)
  • 16:32 twentyafterfour: Restarting apache2 on phab1001 (deploy phabricator hotfix: D876 )
  • 16:23 jynus: stop labsdb1010 mariadb to clone it later to labsdb1009 T179244
  • 15:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1105 in s1 and s2 - T178359 (duration: 00m 45s)
  • 15:00 thcipriani@tin: testing IRC logging
  • 15:00 urandom: Decommissioning Cassandra, restbase1012-c.eqiad.wmnet (T179422)
  • 14:37 moritzm: installing postgres security updates on maps*
  • 14:29 zeljkof: EU SWAT finished
  • 14:07 moritzm: installing postgres security updates on labsdb1004
  • 13:11 moritzm: installing openssl updates
  • 13:10 akosiaris: shutdown install1002 for disk resize
  • 12:57 godog: upgrade grafana to 4.6.1 on https://grafana-labs.wikimedia.org/ - T180428
  • 12:03 ema: restart varnish-be on cp3007, requests failing with 'no backend connection'
  • 11:49 moritzm: rebooting remaining app servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 11:22 moritzm: rebooting remaining API servers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 11:04 godog: reimage restbase2002 - T179422
  • 10:50 marostegui: Deploy alter table on s5: db1104 db1100 db1106 - T174569
  • 10:32 elukey: removed old target configs from /srv/prometheus/analytics/targets on prometheus100[34] after https://gerrit.wikimedia.org/r/391179
  • 10:23 moritzm: rebooting image scalers in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 10:07 godog: upload scap 3.7.2-1 - T127762
  • 09:59 mobrovac@tin: Synchronized wmf-config/jobqueue.php: JobQueue: Migrate RecordLintJob to EventBus - T175212 (duration: 00m 46s)
  • 09:56 marostegui: Deploy alter table on s5 - dbstore1001 - T174569
  • 09:49 foks: Disabled 2FA for Jean-Frédéric
  • 09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 46s)
  • 09:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1105 as multi-instance host for s1 and s2 - T178359 (duration: 00m 47s)
  • 09:43 godog: upgrade prometheus to 1.8.1 with k8s on prometheus2004 - T177395
  • 09:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing (duration: 00m 40s)
  • 09:27 ppchelko@tin: Started deploy [cpjobqueue/deploy@df5fca9]: Enable RecordLintJob processing
  • 09:07 moritzm: rebooting remaining app servers in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2108.codfw.wmnet
  • 09:05 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: wtp2018.codfw.wmnet
  • 08:31 marostegui: Deploy alter table on s5 - dbstore1002 - T174569
  • 08:22 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list T180147 PT 2/2 LABS ONLY (duration: 00m 46s)
  • 08:21 addshore@tin: Synchronized wmf-config/extension-list: Add AdvancedSearch to extension-list T180147 PT 1/2 (duration: 00m 47s)
  • 08:13 moritzm: rebooting job runners in codfw for update to Linux 4.9.51 (and to pick up OpenSSL updates)
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
  • 06:18 marostegui: Deploy alter table on s6 primary master (db1061) - T174569
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 14 02:30:59 UTC 2017 (duration 6m 38s)
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 07m 28s)
  • 01:35 mdholloway: mobileapps finished rolling back to 11/8 deployment
  • 01:34 mholloway-shell@tin: Finished deploy [mobileapps/deploy@00e60b2]: (no justification provided) (duration: 02m 27s)
  • 01:31 mholloway-shell@tin: Started deploy [mobileapps/deploy@00e60b2]: (no justification provided)
  • 01:30 mdholloway: mobileapps rolling back today's deployment due to a significant increase in errors.

2017-11-13

  • 22:49 bawolff: deployed patch T119158 (Will affect language converter. -{}- no longer allowed in link urls)
  • 22:42 bawolff: deployed patch T124404
  • 22:02 mholloway-shell@tin: Finished deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862 (duration: 05m 31s)
  • 21:57 mholloway-shell@tin: Started deploy [mobileapps/deploy@9b10959]: Update mobileapps to c002862
  • 21:40 urandom: Decommissioning Cassandra, restbase1012-b.eqiad.wmnet (T179422)
  • 20:20 ejegg: updated CiviCRM from ddc3881 to f571c67
  • 19:20 smalyshev@tin: Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 20s)
  • 19:20 smalyshev@tin: Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593
  • 19:18 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable per-filter profiling on enwiki T179323 (duration: 00m 45s)
  • 19:16 thcipriani@tin: Synchronized docroot/search.wikimedia.org/index.php: search.wikimedia.org: Clean up result returning logic (duration: 00m 47s)
  • 19:15 madhuvishy: Kick of second dumps rsync from ms1001 to labstore1006
  • 18:28 gehel: deploy latest blazegraph + GUI on wdqs200[23] to switch vocabulary - T176593
  • 18:20 elukey: drain + shutdown analytics1029 as prep step to replace the BBU - T178742
  • 18:09 hoo: Ran "scap pull" on mwdebug1001/snapshot1001 after (further) tests re T177486
  • 18:02 urandom: Restarting Cassandra, restbase-dev1004.eqiad.wmnet (testing new `c-foreach-restart`)
  • 18:00 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanup (duration: 00m 47s)
  • 17:28 hoo: Ran "scap pull" on mwdebug1001 after tests re T177486
  • 17:12 mobrovac: restbase depooling restbase1007, restbase1012, restbase1014, restbase2002, restbase2004, restbase2006 for T179422
  • 16:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 47s)
  • 16:45 mobrovac@tin: Finished deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396 (duration: 03m 45s)
  • 16:42 mobrovac@tin: Started deploy [mathoid/deploy@63b2ddc]: Update to service-template-node v0.5.3 - T151396
  • 16:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174569 (duration: 00m 46s)
  • 15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp2017.codfw.wmnet
  • 15:58 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp2017.codfw.wmnet
  • 15:22 otto@tin: Finished deploy [eventlogging/analytics@e024af3]: T179625 (duration: 00m 02s)
  • 15:22 otto@tin: Started deploy [eventlogging/analytics@e024af3]: T179625
  • 15:09 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>. (duration: 00m 02s)
  • 15:09 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting, got an error: userAgent is a <type unicode>.
  • 15:08 urandom: Decommissioning Cassandra, restbase1012-a.eqiad.wmnet (T179422)
  • 15:06 otto@tin: Finished deploy [eventlogging/analytics@5796c27]: T179625 (duration: 00m 04s)
  • 15:06 otto@tin: Started deploy [eventlogging/analytics@5796c27]: T179625
  • 14:58 herron: upgrading puppetmaster2002 to puppet 4
  • 14:15 zeljkof: EU SWAT finished
  • 14:10 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist jenkins in test wiki (T167432) (duration: 00m 47s)
  • 14:08 marostegui: Deploy alter table on db1102.s6 (with replication - sanitarium master) - T174569
  • 13:49 marostegui: Stop replication on labsdb1010 to copy cebwiki.geo_tags table
  • 13:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
  • 13:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Increase weight for db1103 on s2 and s4 - T178359 (duration: 00m 46s)
  • 13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174569 (duration: 00m 46s)
  • 13:09 marostegui: Deploy schema change on db1083 - T174569
  • 12:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174569 (duration: 00m 47s)
  • 12:14 ema: cache_misc: upgrade varnish to 5.1.3-1wm2
  • 12:08 ema: cp3008: upgrade varnish to 5.1.3-1wm2
  • 12:02 ema: cp4021: restart varnish-be (mbox lag)
  • 11:56 moritzm: installing imagemagick security updates
  • 11:48 moritzm: installing irssi security updates
  • 11:18 elukey: restart of all the druid daemons on druid100[1-6] to apply the new prometheus jmx jvm exporters - T177459
  • 10:57 gehel: upgrade elasticsearch on cirrus / codfw - T178411
  • 10:49 marostegui: Deploy schema change on db1088 - T174569
  • 10:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174569 (duration: 00m 54s)
  • 10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174569 (duration: 00m 46s)
  • 10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 46s)
  • 10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 as multi-instance host for s2 and s4 - T178359 (duration: 00m 47s)
  • 10:12 moritzm: rebooting remaining Parsoid hosts in eqiad for update to Linux 4.9.51 (and to pick up OpenSSL update)
  • 09:44 godog: test upgrade of prometheus 1.8.1 with k8s on prometheus2003 - T177395
  • 09:29 moritzm: rebooting mw1221-mw1235 for update to Linux 4.9.51 (and to pick up OpenSSL update)
  • 09:02 elukey: restart of druid brokers on druid100[1-6] to apply https://gerrit.wikimedia.org/r/390419 - T177459
  • 08:41 marostegui: Deploy alter table on db1093 - T174569
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174569 (duration: 00m 46s)
  • 08:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 after alter table - T174569 (duration: 00m 47s)
  • 08:33 moritzm: rebooting mw1238-mw1258 for update to Linux 4.9.51 (and to pick up OpenSSL update)
  • 07:48 moritzm: installing ruby2.3 security updates
  • 07:38 marostegui: Deploy alter table to db1104 - T179106
  • 07:33 marostegui: Optimize wb_terms table on db2052 - T179106
  • 06:44 marostegui: Deploy alter table directly on codfw s5 master (db2023), this will generate lag on codfw - T179793
  • 06:27 marostegui: Deploy alter table on db1098 - T174569
  • 06:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174569 (duration: 00m 49s)
  • 06:18 marostegui: Deploy alter table db2086 - T179106
  • 04:01 urandom: Decommissioning Cassandra, restbase1007-c.eqiad.wmnet (T179422)
  • 02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 13 02:38:38 UTC 2017 (duration 6m 42s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 08m 18s)

2017-11-12

  • 19:39 urandom: Decommissioning Cassandra, restbase1007-b.eqiad.wmnet (T179422)
  • 16:19 bblack: cp4026 - restart backend (mailbox lag)
  • 01:29 urandom: Decommissioning Cassandra, restbase1007-a.eqiad.wmnet (T179422)

2017-11-11

  • 13:52 urandom: Decommissioning Cassandra, restbase2006-c.codfw.wmnet (T179422)
  • 00:48 urandom: Decommissioning Cassandra, restbase2006-b.codfw.wmnet (T179422)

2017-11-10

  • 18:08 smalyshev@tin: Finished deploy [wdqs/wdqs@ccab8ce]: data reload/T176593 (duration: 00m 34s)
  • 18:08 smalyshev@tin: Started deploy [wdqs/wdqs@ccab8ce]: data reload/T176593
  • 17:20 moritzm: uploaded icu 57.1-6+wmf1 for jessie-wikimedia/component/icu57 (co-installable build for ICU migration)
  • 17:11 moritzm: freed some disk space on install1002
  • 16:47 marostegui: Deploy alter table on s6, dbstore1001, dbstore1002 abd db1030 - T174569
  • 15:03 moritzm: rebooting remaing API servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 14:57 urandom: Decommissioning Cassandra, restbase2006-a.codfw.wmnet (T179422)
  • 14:17 moritzm: rebooting image scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 13:10 marostegui: truncate /var/log/nginx/error.log.1 on install1002 as it is filling up
  • 13:09 moritzm: powercycling mw2118, stuck after reboot
  • 13:05 ema: cp4021: restart varnish-be due to mbox lag
  • 12:48 moritzm: rebooting video scalers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 11:46 _joe_: restarted all services and repooled scb1001
  • 11:38 moritzm: rebooting mw2163-2199 to 4.9.51 (and to pick up OpenSSL updates)
  • 11:32 _joe_: stopping mobileapps as well on scb1001
  • 11:27 moritzm: rebooting wtp1025 to 4.9.51
  • 11:22 _joe_: stopping changeprop, celery-ores, cpjobqueue on scb1001
  • 11:21 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Disable AdvancedSearch on deployment.beta BETA ONLY T180201 (duration: 00m 46s)
  • 11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old comment about db1080 (duration: 00m 46s)
  • 11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T178359 (duration: 00m 46s)
  • 10:59 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: wtp2017.codfw.wmnet
  • 10:55 _joe_: depooling scb1001 from all services while it becomes healthy again
  • 10:52 _joe_: restarting ores on scb1001, causing memory exhaustion
  • 10:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 47s)
  • 10:49 moritzm: powercycling wtp2017, stuck after reboot
  • 10:24 marostegui: Deploy schema change on db2089 - T179106
  • 10:11 moritzm: rebooting Parsoid servers in codfw to 4.9.51 (and to pick up OpenSSL updates)
  • 10:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight - T178359 (duration: 00m 58s)
  • 09:54 marostegui: Compress enwiki on db1105.s1 - T178359
  • 09:45 moritzm: powercycling mw2108, stuck after reboot
  • 09:30 hashar: Upgrading operations-puppet-tests-docker jenkins job to stop passing docker --tty and thus have signals forwarded from 'docker run' - T176747
  • 09:23 moritzm: rebooting mw2097-mw2117 to 4.9.51 (and to pick up OpenSSL updates)
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 with low weight - T178359 (duration: 00m 47s)
  • 09:13 moritzm: powercycling mw2213, stuck after reboot
  • 08:43 moritzm: rebooting mw2200-mw2223 to 4.9.51 (and to pick up OpenSSL updates)
  • 07:50 smalyshev@tin: Finished deploy [wdqs/wdqs@213f864]: (no justification provided) (duration: 00m 33s)
  • 07:49 smalyshev@tin: Started deploy [wdqs/wdqs@213f864]: (no justification provided)
  • 07:27 _joe_: restarting apache on phab1001
  • 07:20 marostegui: Deploy alter table on s3.codfw master (db2018) with replication, this will generate lag on codfw - T174569
  • 06:50 marostegui: Deploy alter table on s5 eqiad master (db1063) - T172207
  • 06:41 marostegui: Stop MySQL on db1055 to copy its content to db1105 - T178359
  • 06:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 (duration: 00m 49s)
  • 06:39 marostegui: Force a BBU relearn on db1046 - T166141
  • 01:21 aaron@tin: Synchronized php-1.31.0-wmf.7/includes/db: Use the main stash for LBFactory "memStash" parameter (duration: 00m 47s)
  • 01:18 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: minor cleanups, less 500s (duration: 00m 47s)

2017-11-09

  • 22:28 urandom: Decommissioning Cassandra, restbase2004-c.codfw.wmnet (T179422)
  • 20:28 demon@tin: Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: less spammy error logs (duration: 00m 47s)
  • 20:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.7
  • 19:06 ebernhardson@tin: Synchronized php-1.31.0-wmf.7/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off DBN sizing AB test (duration: 00m 51s)
  • 19:00 bblack: cp3030 - end experimentation, puppetizing back to normal config
  • 18:46 bblack: cp3030 - round 2 of ssl_do_wait_shutdown test
  • 18:41 arlolra: Updated Parsoid to 2887b5ad (T178253, T173643, T176728, T180010, T171381, T179757)
  • 18:28 arlolra@tin: Finished deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad (duration: 12m 20s)
  • 18:22 bblack: cp3030: puppet-disabled + manual nginx ssl_do_wait_shutdown config
  • 18:16 arlolra@tin: Started deploy [parsoid/deploy@d1c7386]: Updating Parsoid to 2887b5ad
  • 18:14 moritzm: not rebooting parsoid hosts due to Services deployment window, instead rolling restart of mw2120-mw2139 for kernel update to 4.9.51
  • 18:04 moritzm: rolling restart of parsoid servers in codfw for 4.9.51 kernel update
  • 17:14 urandom: Restarting Cassandra, restbase2005-b.codfw.wmnet (T179419)
  • 16:33 urandom: Restarting Cassandra, restbase2005-a.codfw.wmnet (T179419)
  • 15:12 urandom: Creating mathoid schema (T179419)
  • 15:05 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: Enable AdvancedSearch on beta LABS / BETA ONLY PT2/2 (duration: 00m 49s)
  • 15:04 addshore: last sync was actually "Enable AdvancedSearch on beta"
  • 15:04 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY PT1/2 (duration: 00m 50s)
  • 14:57 addshore@tin: Synchronized wmf-config/extension-list-labs: Add AdvancedSearch to extension-list-labs LABS / BETA ONLY (duration: 00m 50s)
  • 14:42 zeljkof: EU SWAT finished
  • 14:40 zfilipin@tin: Synchronized wmf-config/StartProfiler.php: SWAT: xenon: encode the request method as a virtual stack frame (duration: 00m 50s)
  • 14:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Use a threshold that ores in frwiki can stand (T180115) (duration: 00m 50s)
  • 14:30 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/ContentTranslation/modules/: SWAT: Bring back the overlay support for a specific screen region (T179997) (duration: 00m 50s)
  • 14:25 zfilipin@tin: Synchronized php-1.31.0-wmf.7/extensions/EventBus/EventBus.php: SWAT: Logging improvements Rename logged field to fix logstash mapping (duration: 00m 54s)
  • 14:09 urandom: Decommissioning Cassandra, restbase2004-b.codfw.wmnet (T179422)
  • 13:04 moritzm: rebooting mw1209-mw1220 (app servers) to 4.9.51 (also to pick up new OpenSSL)
  • 12:38 moritzm: rebooting mw1189-mw1208 (API servers) to 4.9.5 (also to pick up new OpenSSL)
  • 12:37 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.6$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=frwiki (T180115)
  • 11:51 moritzm: rebooting mw1180-mw1188 (app servers) to 4.9.5 (also to pick up new OpenSSL)
  • 11:45 _joe_: removed all local hacks from puppetmaster1001, now it uses rhodium again
  • 11:37 _joe_: cleaning up spurious directories /var/lib/puppet/server/ssl/ca from eqiad's puppetmaster backends, generated due to some error on 8/11/2017
  • 11:10 ema: powercycle cp2008, stuck rebooting
  • 11:03 moritzm: rebooting mw1276-mw1279 (API canaries) to 4.9.5 (also to pick up new OpenSSL)
  • 09:50 moritzm: rolling reboot of scb in eqiad for kernel update (also to pick up openssl updates)
  • 07:56 ema: cp1074 failed rebooting, power-cycled
  • 07:46 _joe_: restarting apache on rhodium after setting --profile --trace in the puppet settings
  • 07:04 legoktm@tin: Synchronized php-1.31.0-wmf.7/resources/: Restore jquery.badge and jquery.placeholder modules (duration: 00m 53s)
  • 02:58 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 9 02:58:30 UTC 2017 (duration 6m 59s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 09m 09s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 09m 18s)
  • 01:02 twentyafterfour: Not deploying any phabricator updates this week.

2017-11-08

  • 23:45 urandom: Decommissioning Cassandra, restbase2004.codfw.wmnet (T179422)
  • 22:50 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7 try #2
  • 22:43 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/CirrusSearch/: Backport cirrus rescore profile refactor to wmf.6 (duration: 01m 02s)
  • 22:21 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to wmf.6
  • 22:09 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.7
  • 21:44 bsitzmann@tin: Finished deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 (T178706 T178708 T178333 T170692) (duration: 07m 12s)
  • 21:37 bsitzmann@tin: Started deploy [mobileapps/deploy@00e60b2]: Update mobileapps to 8e82983 (T178706 T178708 T178333 T170692)
  • 21:34 ejegg: re-started donations queue consumer with new thank you letter
  • 21:09 ejegg: updated CiviCRM from b11c591 to ddc3881
  • 20:52 ejegg: turned off donations consumer for ty letter update
  • 20:36 ejegg: updated payments-wiki from a539d27 to d150287
  • 20:33 aaron@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralAuth: Use the proper cache key method in loadFromCache() (duration: 00m 54s)
  • 20:22 apergos: rsync from ms1001 to labstore1006 of dumps, 17T so expect it to take several days
  • 20:10 herron: depooled rhodium via puppetmaster1001 apache config
  • 20:09 demon@tin: Synchronized php: symlink (duration: 00m 49s)
  • 19:57 demon@tin: Synchronized php-1.31.0-wmf.7/extensions/CentralAuth/includes/CentralAuthUser.php: (no justification provided) (duration: 00m 51s)
  • 19:55 urandom: Creating mathoid schema (T179419)
  • 19:31 niharika29@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
  • 19:31 urandom: Creating page restrictions schema (T179421)
  • 19:30 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/ORES/: Store stats of accessing ores service for getting thresholds T179862 (duration: 00m 51s)
  • 19:24 niharika29@tin: Synchronized php-1.31.0-wmf.7/extensions/CirrusSearch/: Revert Improve handling of 5xx responses to elasticsearch requests (duration: 01m 02s)
  • 19:14 smalyshev@tin: Finished deploy [wdqs/wdqs@b330bc8]: Update service whitelist (duration: 02m 58s)
  • 19:11 smalyshev@tin: Started deploy [wdqs/wdqs@b330bc8]: Update service whitelist
  • 18:27 demon@tin: Synchronized wmf-config/: Dropping old PrivateSettings symlink (ducks and covers) (duration: 00m 52s)
  • 18:19 demon@tin: Synchronized phpcs.xml: no-op (duration: 00m 50s)
  • 17:58 urandom: Restarting Cassandra, restbase2005-[abc]
  • 17:51 urandom: Clearing snapshots in RESTBase legacy Cassandra cluster (T179417)
  • 17:47 marostegui: Deploy alter table on s1 codfw primary master (db2048) with replication, this will generate lag on codfw - T174569
  • 17:43 mobrovac: restbase truncate the default parsoid storage group's tables for T179417
  • 17:41 urandom: Restarting Cassandra, restbase2001-[abc]
  • 17:19 urandom: Restarting Cassandra, restbase2003-[abc]
  • 17:09 herron: regenerated rhodium puppet certificate
  • 17:08 urandom: Restarting Cassandra, restbase1009-[abc]
  • 16:58 urandom: Restarting Cassandra, restbase1008-[abc]
  • 16:49 awight@tin: Finished deploy [ores/deploy@82a13ae]: Roll back scb1002 (duration: 02m 37s)
  • 16:47 awight@tin: Started deploy [ores/deploy@82a13ae]: Roll back scb1002
  • 16:45 awight@tin: Finished deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1 (duration: 05m 45s)
  • 16:40 awight@tin: Started deploy [ores/deploy@1b0e59f]: Try to purge specter of revscoring 1
  • 16:37 urandom: Restarting Cassandra, restbase1010-[abc]
  • 16:37 godog: disregard message about thumbor rolling-restart, upgrade already done and only thumbor1001 rebooted now
  • 16:34 awight@tin: Finished deploy [ores/deploy@82a13ae]: Fix ORES on scb1002 (duration: 00m 03s)
  • 16:34 awight@tin: Started deploy [ores/deploy@82a13ae]: Fix ORES on scb1002
  • 16:29 godog: roll-restart thumbor in eqiad for kernel upgrade
  • 16:11 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420 (duration: 07m 22s)
  • 16:04 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2b - T179420
  • 16:02 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420 (duration: 00m 13s)
  • 16:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage, take 2 - T179420
  • 16:01 otto@tin: Finished deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early (duration: 00m 02s)
  • 16:01 otto@tin: Started deploy [eventlogging/analytics@03285e4]: Reverting EvenCapsule update and fixes, processes got restarted too early
  • 15:33 urandom: Decommissioning restbase2001-c.codfw.wmnet (T179422)
  • 15:23 _joe_: testing changes on rhodium regarding hostprivkey,hostcert
  • 15:21 ema: eqiad lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 15:17 otto@tin: Finished deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625 (duration: 00m 04s)
  • 15:17 otto@tin: Started deploy [eventlogging/analytics@02c5a6b]: EventCapsule update and fixes, this is no-op as is. T179625
  • 15:01 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16 on all targets: T180017
  • 14:55 otto@tin: Started restart [eventlogging/eventbus@41e3418]: Bumping worker processes to 16: T180017
  • 14:54 ema: esams lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 14:54 otto@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
  • 14:54 otto@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 14:41 ema: powercycle cp1050 (failed reboot)
  • 14:29 addshore@tin: Synchronized php-1.31.0-wmf.7/docs/uidesign/mediawiki.diff.html: SWAT Add render moved paragraphs marker in diff view PT 2/2 DOCS ONLY (duration: 00m 50s)
  • 14:28 addshore@tin: Synchronized php-1.31.0-wmf.7/resources/src/mediawiki/mediawiki.diff.styles.css: SWAT Add render moved paragraphs marker in diff view PT 1/2 (duration: 00m 51s)
  • 14:16 volans: upgrading cumin to v1.3.0 on prod and WMCS cumin masters
  • 14:10 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: SWAT: Logging improvements. (duration: 00m 52s)
  • 14:08 ema: codfw lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 13:03 hasharAway: Upgrading jenkins on contint1001/contint2001
  • 12:54 bblack@puppetmaster1001: conftool action : set/pooled=yes; selector: name=phab2001-vcs.codfw.wmnet
  • 12:53 bblack: restart pybal on lvs2002 for git-ssh.codfw deploy
  • 12:51 bblack: restart pybal on lvs2005 for git-ssh.codfw deploy
  • 12:46 mutante: osmium - re-enabling puppet - temp test is over and will be decom'ed
  • 12:41 mutante: krypton (misc PHP apps, scholarships.wm, iegreview.wm, grafana, racktables, burrow) rebooting for kernel upgrade
  • 12:35 mutante: bromine (misc static tistes, annual/transparency/static-bz) - rebooting for kernel upgrade
  • 12:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1051 original weight after maintenance - T178359 (duration: 00m 50s)
  • 12:03 mutante: alcyone (url-downloader) rebooting for kernel upgrade
  • 11:53 mutante: netmon1003 (servermon) - rebooting for kernel upgrade
  • 11:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
  • 11:48 moritzm: installed openjdk-8/openssl updates and new kernels on restbase*
  • 11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1051 weight after maintenance - T178359 (duration: 00m 50s)
  • 11:12 moritzm: installing jenkins security update on releases*
  • 11:10 moritzm: imported jenkins 2.73.3 to apt.wikimedia.org
  • 10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 with low weight after maintenance - T178359 (duration: 01m 01s)
  • 10:57 ema: ulsfo lvs reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m
  • 10:42 mutante: actinium - reboot for kernel upgrade (url-downloader)
  • 10:40 ema: varnish 5.1.3-1wm2 built and uploaded to apt.w.o (experimental)
  • 10:39 mutante: aluminium - reboot for kernel upgrade (url-downloader)
  • 10:28 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1005.eqiad.wmnet
  • 10:18 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet
  • 10:07 elukey: reboot aqs100[4-9] for jvm and kernel updates
  • 09:37 mutante: planet1001, alsafi (url-downloader) - apt autoremove; reboot for kernel upgrade
  • 09:32 mutante: restarting ircecho (icinga-wm)
  • 09:26 mutante: ununpentium (rt.wikimedia.org): apt-get autoremove; reboot for kernel upgrade
  • 09:23 mutante: rutherforidum (people.wikimedia.org) : apt-get autoremove ; reboot for kernel upgrade
  • 09:22 Pchelolo: restart cassandra-a on restbase1010
  • 09:09 Pchelolo: restart restbase on 1013 and 1015
  • 08:58 mutante: planet2001 - apt autoremove; reboot for kernel upgrade
  • 08:40 ema: resume cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 07:55 marostegui: Deploy alter table on s1 - on codfw master (db2048) with replication enabled - T172207
  • 07:46 marostegui: Deploy alter table on s2 - on codfw master (db2017) with replication enabled - T172207
  • 07:21 marostegui: Deploy alter table on s3 - on codfw master (db2018) with replication enabled - T172207
  • 07:13 marostegui: Deploy alter table on s7 - on codfw master (db2029) with replication enabled - T172207
  • 07:13 krinkle@tin: Synchronized docroot/noc/conf/: I2e51e783a (duration: 01m 06s)
  • 06:57 marostegui: Deploy alter table on s6 - on codfw master (db2028) with replication enabled - T172207
  • 06:26 marostegui: Add 330G to db2023 partition to make sure the alter over logging table runs fine - T174569
  • 06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite2001
  • 06:16 Krinkle: Restarted uwsgi-graphite-web service on graphite1001
  • 06:15 marostegui: Stop MySQL on db1051 to copy its content to db1105.s1 - T178359
  • 06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T178359 (duration: 00m 51s)
  • 05:45 Krinkle: rm -rf /var/lib/carbon/whisper/MediaWiki/wanobjectcache/centralauth_user_* on graphite1001 and graphite2001 for T179999
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 8 03:17:53 UTC 2017 (duration 7m 13s)
  • 03:10 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.7) (duration: 15m 16s)
  • 02:44 aaron@tin: Synchronized php-1.31.0-wmf.7/tests: Deploy 087f2d579a9f which reverts 4432e898be0 due to statsd spam (duration: 01m 14s)
  • 02:42 aaron@tin: Synchronized php-1.31.0-wmf.7/includes: Deploy 087f2d579a9f which reverts 4432e898be0 due to statsd spam (duration: 01m 40s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 38s)

2017-11-07

  • 23:57 urandom: Decommissioning restbase2001-b.codfw.wmnet (T179422)
  • 22:50 ejegg: re-enabled thank you mailer
  • 22:49 ejegg: updated CiviCRM from 85e89c6 to dc1b279
  • 22:39 hoo: Reset a global account's email, per T179950
  • 22:28 ejegg: updated CiviCRM from 122ba65 to 85e89c6
  • 21:56 ejegg: updated CiviCRM from dc9d054 to 122ba65
  • 21:20 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: T176948 Remove Shared Cache settings from Wikibase-buildentry (duration: 00m 50s)
  • 21:16 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Stop using wgWikibaseSharedCacheKeyPrefix from Wikidata build (duration: 00m 49s)
  • 21:09 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 3/3 (LABS) (duration: 00m 50s)
  • 21:07 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 2/3 (duration: 00m 52s)
  • 21:06 addshore@tin: Synchronized wmf-config/Wikibase.php: T176948 Remove wmgWikibaseUseConfigFromWikidataBuild PT 1/3 (duration: 00m 50s)
  • 21:02 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 wmgWikibaseUseConfigFromWikidataBuild flase for all of PROD (duration: 00m 49s)
  • 21:01 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidataclient (duration: 00m 50s)
  • 20:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for group0 & group1 (duration: 00m 50s)
  • 20:54 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for hewiki (duration: 00m 49s)
  • 20:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T176948 Load wikibase build from mediawiki-config for wikidatawiki (duration: 00m 50s)
  • 20:48 herron: puppet issue cleared after reverting 386666. restarting ircecho on einsteinium
  • 20:45 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T176948 #1 #2 #3 Load wikibase build from mediawiki-config for BETA ONLY (duration: 00m 50s)
  • 20:28 addshore@tin: Synchronized wmf-config/Wikibase.php: Add loading of wikibase extensions from build PT 3/3 (duration: 00m 50s)
  • 20:27 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add loading of wikibase extensions from build PT 2/3 (duration: 00m 50s)
  • 20:25 addshore@tin: Synchronized wmf-config/Wikibase-buildentry.php: Add loading of wikibase extensions from build PT 1/3 (duration: 00m 49s)
  • 20:20 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 2/2 (duration: 00m 50s)
  • 20:19 addshore@tin: Synchronized wmf-config/CommonSettings.php: Remove unused wmgUseWikibasePropertySuggester PT 1/2 (duration: 00m 50s)
  • 20:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.7
  • 19:44 volans: restarted apache2 on rhodium (puppet master failing)
  • 19:31 herron: restart apache2 service on rhodium
  • 19:12 demon@tin: Finished scap: wmf.7 bootstrap (duration: 48m 15s)
  • 19:03 awight: begin stress test on ores*
  • 18:58 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production) (duration: 00m 31s)
  • 18:57 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1009 (non-production)
  • 18:56 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production) (duration: 00m 32s)
  • 18:56 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1008 (non-production)
  • 18:50 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production) (duration: 00m 33s)
  • 18:49 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to repair ores1002 (non-production)
  • 18:39 volans: slowly running puppet on failed hosts with cumin (concurrency=5)
  • 18:38 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 00m 20s)
  • 18:38 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
  • 18:36 _joe_: restarting apache2 on rhodium
  • 18:24 demon@tin: Started scap: wmf.7 bootstrap
  • 18:16 awight@tin: Finished deploy [ores/deploy@29905e5]: test deployment to ores* (non-production) (duration: 01m 04s)
  • 18:15 awight@tin: Started deploy [ores/deploy@29905e5]: test deployment to ores* (non-production)
  • 18:06 herron: restarting puppetdb service on nitrogen
  • 17:56 urandom: Decommissioning restbase2001-a.codfw.wmnet (T179422)
  • 17:55 elukey: stop ircecho on einstenium (puppet shower from nitrogen)
  • 17:40 ema: stop cache_text/upload rolling reboots, resuming tomorrow
  • {{safesubst:SAL entry|1=17:38 urandom: Restart Cassandra, restbase1010-{a,b,c}.eqiad.wmnet (T178177)}}
  • 17:33 urandom: Clearing Cassandra snapshots (T179422)
  • 17:32 moritzm: rolling reboot of scb in codfw to pick up new kernel (and openssl updates)
  • {{safesubst:SAL entry|1=17:02 urandom: Restarting Cassandra, restbase2001-{a,b,c} to apply OpenJDK upgrade}}
  • 16:47 demon@tin: Synchronized multiversion/submodules.json: no op (duration: 00m 47s)
  • 16:12 ema: start cache_text/upload rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 15:46 gehel: rolling restart of cassandra maps-test for logging change
  • 15:43 godog: roll-restart thumbor in eqiad for kernel upgrade
  • 15:37 urandom: T179420: recreating wiktionary definition schemas
  • 15:29 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420 (duration: 06m 46s)
  • 15:22 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: revert definition switch, wrong schema - T179420
  • 15:12 _joe_: added a runner for htmlCacheUpdate on cewiki too
  • 15:10 mobrovac@tin: Finished deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420 (duration: 07m 52s)
  • 15:02 mobrovac@tin: Started deploy [restbase/deploy@c5dd1e2]: Switch wiktionary definitions to use the next-gen storage - T179420
  • 14:52 godog: roll-restart thumbor for kernel upgrades
  • 14:38 mobrovac: restbase creating wiktionary definition schemas for T179420
  • 14:36 godog: reboot wezen for kernel upgrade
  • 14:33 godog: reboot lithium for kernel upgrade
  • 14:26 elukey: rolling restart of kafka on kafka-jumbo* for jvm security updates
  • 14:20 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
  • 14:19 moritzm: rearming keyholder on naos
  • 14:14 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ShortUrl on pa.wiki - T178919 (duration: 00m 46s)
  • 14:11 hashar: mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=pawiki ShortUrl - T178919
  • 14:08 moritzm: rebooting tureis for kernel update
  • 14:07 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: add _ in appendix talk namespace at mywiktionary - T179907 (duration: 00m 45s)
  • 14:02 moritzm: actually holding mw canary reboots until SWAT is over
  • 14:01 moritzm: rebooting mw canaries to 4.9.51 kernel (also picking up openssl/openssl1.1 updates)
  • 13:50 herron: rebooted fermium (lists) for kernel update
  • 13:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment about db1105 current status (duration: 00m 47s)
  • 13:43 herron: starting rolling mx reboots for kernel update
  • 13:31 marostegui: Deploy schema change on s5 codfw master (db2023) with replication, this will generate lag on codfw - T174569
  • 13:19 moritzm: rebooting naos for kernel update
  • 13:17 gehel: reboot maps eqiad cluster for upgrades
  • 13:10 moritzm: rebooting wasat for kernel update
  • 12:50 mutante: hafnium, tungsten: groupdel perf-roots to go with gerrit:389663 (T179728)
  • 12:00 mobrovac@tin: Finished deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417 (duration: 08m 14s)
  • 11:52 mobrovac@tin: Started deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417
  • 11:43 mobrovac: restbase truncating cassandra 2 non-WP tables for T179417
  • 11:42 mobrovac: restbase truncating cassandra 2 non-WP tables for T179420
  • 11:29 moritzm: installing java security updates/restarting cassandra on restbase2001 (cassandra3 node)
  • 11:07 ema: cache_misc rolling reboots: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 10:59 ema: reboot cp3007: upgrading kernel to 4.9.51, libssl to 1.0.2m and 1.1.0g
  • 10:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
  • 10:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2092 as recentchanges multi-instance host on s1 and s3 T178359 (duration: 00m 45s)
  • 10:41 moritzm: restarting ntpd on dns recursors to pick up openssl update
  • 10:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 45s)
  • 10:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2091 as recentchanges multi-instance host on s2 and s4 T178359 (duration: 00m 46s)
  • 10:24 elukey: create staging database on db1108 (researchers scratch pad) - T177405
  • 10:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
  • 10:09 gehel: reboot of maps codfw cluster for upgrades
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2086 as recentchanges multi-instance host on s5 and s7 T178359 (duration: 00m 45s)
  • 09:58 paravoid: updating certspotter to 0.5 in apt and tegmen/einsteinium
  • 09:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 46s)
  • 09:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2089 as recentchanges multi-instance host on s6 and s5(s8) T178359 (duration: 00m 45s)
  • 09:13 marostegui: Stop MySQL on db1038 - host to be decommissioned - T177911
  • 09:11 moritzm: installing java security updates/restarting cassandra on restbase2002
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1038 from config - T177911 (duration: 00m 45s)
  • 09:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner (duration: 00m 49s)
  • 09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1038 from config - T177911 (duration: 00m 47s)
  • 09:01 ppchelko@tin: Started deploy [cpjobqueue/deploy@3db0cc4]: Do not set Host header for requests to jobrunner
  • 08:49 marostegui: Run redact_sanitarium for hifwiktionary on db1095 (sanitarium) - T173647
  • 08:48 marostegui: Optimize pagelinks and templatelinks on s7 master - db1062 - T174509
  • 07:18 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restarting - T174916
  • 07:11 ema: reboot pinkunicorn for kernel (4.9.51) and openssl (1.0.2m) upgrades
  • 06:19 marostegui: Deploy alter table on s7 codfw master (db2029) with replication, this will cause lag in codfw - T174569
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 7 02:29:54 UTC 2017 (duration 6m 39s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 19s)

2017-11-06

  • 22:34 bawolff: Deploy patch for T178451
  • 21:48 arlolra: Updated Parsoid to 6cb77104 (T179579, T175792)
  • 21:42 arlolra@tin: Finished deploy [parsoid/deploy@d1df3c3]: (no justification provided) (duration: 08m 46s)
  • 21:35 awight: Begin stress testing ores* (non-production)
  • 21:33 arlolra@tin: Started deploy [parsoid/deploy@d1df3c3]: (no justification provided)
  • 21:20 XioNoX: re-activating BGP session to telia in ulsfo
  • 21:16 ladsgroup@tin: Finished deploy [ores/deploy@97a1d80]: Deploying early November (T179837) (duration: 10m 47s)
  • 21:05 ladsgroup@tin: Started deploy [ores/deploy@97a1d80]: Deploying early November (T179837)
  • 20:56 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW Map again (duration: 00m 46s)
  • 20:50 marostegui: Deploy alter table on s2 codfw master (db2017) with replication, this will generate lag in codfw - T174569
  • 20:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hifwiktionary T173643 (duration: 00m 45s)
  • 20:34 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hifwiktionary T173643
  • 20:33 reedy@tin: Synchronized dblists/: hifwiktionary T173643 (duration: 00m 47s)
  • 20:04 reedy@tin: Synchronized wmf-config/interwiki.php: Update IW map (duration: 00m 45s)
  • 19:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: electcomwiki T174370 (duration: 00m 45s)
  • 19:56 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: electcomwiki T174370
  • 19:55 reedy@tin: Synchronized dblists/: electcomwiki T174370 (duration: 00m 44s)
  • 19:52 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 (duration: 02m 49s)
  • 19:47 demon@tin: Pruned MediaWiki: 1.31.0-wmf.5 [keeping static files] (duration: 01m 38s)
  • 19:22 Amir1: ladsgroup@terbium:/srv/mediawiki-staging/php-1.31.0-wmf.4$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki (T179596)
  • 19:21 kaldari@tin: Synchronized wmf-config/InitialiseSettings.php: Updating InitialiseSettings for draftquality data (duration: 00m 48s)
  • 18:15 XioNoX: deactivating BGP session to telia in ulsfo
  • 18:09 gehel@tin: Finished deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update (duration: 01m 53s)
  • 18:07 gehel@tin: Started deploy [wdqs/wdqs@fa8bb12]: WDQS GUI and Blazegraph update
  • 17:50 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 17m 18s)
  • 17:48 urandom: T179083: Restarting Cassandra instances on restbase2001.codfw.wmnet
  • 17:32 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
  • 16:51 marostegui: Optimize pagelinks and templatelinks on s2 master - db1054 - T174509
  • 16:26 bblack: Switching unified TLS certificate in the US
  • 16:25 awight@tin: Finished deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb* (duration: 16m 10s)
  • 16:09 awight@tin: Started deploy [ores/deploy@82a13ae]: Redeploy ORES master to scb*
  • 15:50 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
  • 15:49 akosiaris@tin: Started deploy [ores/deploy@29905e5]: testing deploy
  • 15:43 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 12m 45s)
  • 15:31 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
  • 15:26 gehel: reboot of maps-test cluster for jvm and kernel upgrades
  • 15:13 ppchelko@tin: Finished deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection (duration: 00m 29s)
  • 15:12 ppchelko@tin: Started deploy [cpjobqueue/deploy@96f55c6]: USe correct regex for job selection
  • 15:03 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 3/3 (duration: 00m 46s)
  • 15:01 addshore@tin: Synchronized wmf-config/CommonSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 2/3 (duration: 00m 46s)
  • 15:00 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT Set wgWikiDiff2MovedParagraphDetectionCutoff for group0 PT 1/3 (duration: 00m 47s)
  • 14:57 gehel: reboot of relforge for kernel + jvm upgrade
  • 14:50 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch MessageIndexRebuildJob, flaggedrevs_CacheUpdate and deleteLinks jobs to the EventBus infrastructure - T175210 (duration: 00m 46s)
  • 14:50 ppchelko@tin: Finished deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210 (duration: 00m 44s)
  • 14:49 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: Start processing all 'hearted' jobs T175210
  • 14:48 ppchelko@tin: Started deploy [cpjobqueue/deploy@e93feba]: (no justification provided)
  • 14:45 awight@tin: Finished deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production) (duration: 05m 43s)
  • 14:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2003.codfw.wmnet
  • 14:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2002.codfw.wmnet
  • 14:40 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2002.codfw.wmnet
  • 14:40 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
  • 14:39 awight@tin: Started deploy [ores/deploy@29905e5]: Force deployment on ores* (non-production)
  • 14:35 awight@tin: Finished deploy [ores/deploy@29905e5]: restart services on ores* (non-production) (duration: 04m 48s)
  • 14:34 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs2001.codfw.wmnet
  • 14:33 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
  • 14:30 awight@tin: Started deploy [ores/deploy@29905e5]: restart services on ores* (non-production)
  • 14:27 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1005.eqiad.wmnet
  • 14:27 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
  • 14:21 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1004.eqiad.wmnet
  • 14:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1003.eqiad.wmnet
  • 14:20 awight@tin: Finished deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production) (duration: 19m 22s)
  • 14:18 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: EventBus: Enable logging - T150106 (duration: 00m 47s)
  • 14:14 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1003.eqiad.wmnet
  • 14:14 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet
  • 14:13 mobrovac@tin: Synchronized php-1.31.0-wmf.6/extensions/EventBus/EventBus.php: EventBus - Improve logging for T150106 (duration: 00m 48s)
  • 14:01 gehel: full restart of wdqs eqiad / codfw for multiple upgrades
  • 14:01 awight@tin: Started deploy [ores/deploy@29905e5]: celery 4 -> ores* (non-production)
  • 13:03 elukey: rolling restart of druid historical daemons on druid100[1-6] to apply https://gerrit.wikimedia.org/r/#/c/389429
  • 12:58 moritzm: installing openssl security updates
  • 12:55 moritzm: restarting apache on netmon to pick up openssl update
  • 12:43 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron stuck, restrting - T174916
  • 12:29 moritzm: installing imagemagick security updates
  • 11:45 oblivian@tin: Started deploy [jobrunner/jobrunner@a20d043]: (no justification provided)
  • 11:44 marostegui: Deploy alter table on s6 codfw master (with replication, so this will generate lag on codfw) - T174569
  • 11:19 marostegui: Compress InnoDB on db1103.s2 - T178359
  • 11:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 47s)
  • 11:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2087 as recentchanges multi-instance host on s6 and s7 T178359 (duration: 00m 48s)
  • 10:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1101 original weight after maintenance - T178359 (duration: 00m 46s)
  • 10:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
  • 10:05 moritzm: uploading linux 4.9.51-1~bpo8+1 for jessie-wikimedia to apt.wikimedia.org
  • 09:57 godog: roll-upgrade swift to 2.10.2 in codfw and roll-restart for kernel upgrade - T177739
  • 09:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
  • 09:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2088 as recentchanges multi-instance host on s1 and s2 T178359 (duration: 00m 46s)
  • 09:37 _joe_: manually running htmlCacheUpdate for commonswiki and ruwiki on terbium, T173710
  • 09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2085 as recentchanges multi-instance host on s3 and s5 T178553 T178359 (duration: 00m 46s)
  • 09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2085 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1101 more weight after maintenance - T178359 (duration: 00m 46s)
  • 09:07 moritzm: removed monitoring/RAID packages from stretch-wikimedia/thirdparty, now in thirdparty/hwraid
  • 08:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 with low weight after maintenance - T178359 (duration: 00m 47s)
  • 06:46 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 as multi-instance core host on eqiad file T178553 T178359 (duration: 00m 46s)
  • 06:38 marostegui: Stop MySQL on db1101 to copy its content to db1103.s4 - T178359
  • 06:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101 - T178359 (duration: 00m 48s)
  • 06:19 marostegui: Optimize pagelinks and templatelinks on s6 master (db1061) - T174509
  • 06:16 marostegui: Deploy alter table on s4 codfw master (with no replication) - T174569
  • 02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 6 02:44:41 UTC 2017 (duration 6m 55s)
  • 02:37 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 07m 17s)

2017-11-04

  • 13:34 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
  • 13:19 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=scb1002.eqiad.wmnet
  • 12:36 mobrovac: trending-edits depool it from scb1002 to investigate
  • 00:18 Krinkle: Finished whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)

2017-11-03

  • 20:28 marostegui: Deploy alter table db2058 - T174569
  • 18:52 Krinkle: Starting whisper-mass-resize for frontend.navtiming on graphite2001 (T179622)
  • 18:51 Krinkle: whisper-mass-resize completed for graphite1001.eqiad.wmnet:/var/lib/carbon/whisper/frontend/navtiming (at 01:34 UTC actually)
  • 17:56 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms/database/DatabaseMysqlBase.php: logging (duration: 00m 47s)
  • 16:17 marostegui: Deploy alter table on db2044 - T174569
  • 16:03 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/api/ApiQueryTranslatorStats.php: fix warnings about bad min() calls (duration: 00m 47s)
  • 15:50 cmjohnson1: powering off labvirt1015 to swap CPU (again)
  • 15:49 akosiaris: T177393 enable RBAC for kubernetes in production and staging
  • 15:35 volans: uploaded cumin_1.3.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 14:19 moritzm: installing java sec updates / cassandra restarts on xenon/praseodymium (cerium already done earlier)
  • 14:16 hashar: restarting Jenkins on contint1001
  • 14:12 hashar: Upgrading packages on contint1001 / contint2001
  • 13:50 moritzm: installing heimdal security updates on trusty
  • 13:35 ppchelko@tin: Started restart [electron-render/deploy@8dd5f13]: Restarting, service is stuck
  • 13:33 Pchelolo: restart electron render service
  • 12:41 moritzm: uploaded openjdk 8u151-b12 to apt.wikimedia.org/jessie-wikimedia
  • 11:45 moritzm: restarting jenkins on releases* to pick up openjdk security update
  • 11:41 mutante: gerrit - restart Apache - apply gerrit:354078 - only listen on service IP
  • 11:12 moritzm: uploaded openssl 1.0.2m-1 for to apt.wikimedia.org
  • 10:46 marostegui: Compress InnoDB on db1103.s4 - T178359
  • 10:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add comment to db1103 status (duration: 00m 47s)
  • 10:42 mutante: cobalt: (gerrit) temp disable puppet - gerrit2001: restart apache, apply gerrit:354078 stop listening on all interfaces
  • 10:39 oblivian@tin: Synchronized wmf-config/CommonSettings.php: Increase concurrency of htmlCacheUpdate jobs T173710 (duration: 00m 48s)
  • 09:09 mobrovac@tin: Finished deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418 (duration: 10m 40s)
  • 08:58 mobrovac@tin: Started deploy [restbase/deploy@a67b4a7]: Use only the new storage for summaries - T179418
  • 08:46 ema: pool cp4021
  • 08:16 elukey: drop CommandInvocation_15243810 and CommandInvocation_15243810_15423246 from analytics dbs (db1046/db1047/db1108/dbstore1002) - data archived on HDFS - T166712
  • 08:12 moritzm: installing openjdk-8 security updates on stretch
  • 07:57 marostegui: Deploy alter table on s4.db2037 - T174569
  • 07:29 elukey: depooled mw1191 and powercycle - host down with 'CPU 1 check errors' in racadm getsel
  • 06:39 marostegui: Stop MySQL on db2089.s5 to copy it to db2085 - T178359
  • 06:25 marostegui: Stop MySQL on db1103 to copy it to dbstore1001 and reimage it - T178359

2017-11-02

  • 23:36 krinkle@tin: Synchronized php-1.31.0-wmf.6/extensions/NavigationTiming/modules/ext.navigationTiming.js: Fix zero values - T178479 (duration: 00m 47s)
  • 23:10 herron: disabled puppet master debug logging on puppetmaster2001 via /usr/share/puppet/rack/puppet-master/config.ru
  • 22:48 volans: cleaned daemon.log from puppet spam logs on puppetmaster2001 (15GB) - cc herron
  • 22:39 halfak@tin: Started deploy [ores/deploy@82a13ae]: T179621
  • 22:02 Krinkle: Mass-resizing Graphite/Whisper files on graphite1001 and graphite2001 for T179622 (frontend.* namespace)
  • 21:53 ayounsi@tin: Finished deploy [librenms/librenms@ad48b4d]: (no justification provided) (duration: 00m 05s)
  • 21:52 ayounsi@tin: Started deploy [librenms/librenms@ad48b4d]: (no justification provided)
  • 21:50 XioNoX: upgrading librenms
  • 21:38 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 20:56 madhuvishy: Powercycling labvirt1015
  • 20:18 chasemp: simulate load for labvirt1015 'sudo cumin "name:labvirt1015stresstest*" "stress-ng --cpu 1 --io 2 --vm 1 --vm-bytes 1G &"'
  • 19:45 mepps: updated payments-wiki from b88b805 to a539d27
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.6
  • 18:20 gehel: restarting blazegraph on wdqs1003
  • 18:13 gehel: restarting stuck updater on wdqs1003
  • 17:55 demon@tin: Synchronized php-1.31.0-wmf.6/includes/Message.php: logging fixes (duration: 00m 48s)
  • 17:53 demon@tin: Synchronized php-1.31.0-wmf.6/includes/libs/rdbms: logging improvements (duration: 00m 54s)
  • 17:47 moritzm: uploaded openssl 1.1.0g-1+wmf1 to jessie-wikimedia
  • 17:42 awight@tin: Finished deploy [ores/deploy@0e54a3c]: revscoring 2, T175180 (duration: 33m 50s)
  • 17:08 awight@tin: Started deploy [ores/deploy@0e54a3c]: revscoring 2, T175180
  • 16:14 ema: restart varnish-be on cp4026 due to mbox lag
  • 16:04 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Use only EventBus for processing updateBetaFeatureUserCount - T175210 (duration: 00m 51s)
  • 13:46 chasemp: ugrade dnsmasq-base on labcontrol*
  • 13:36 zeljkof: EU SWAT finished
  • 13:35 zfilipin@tin: Synchronized php-1.31.0-wmf.6/extensions/ContentTranslation/modules/publish/ext.cx.publish.js: SWAT: CX1: Check for template adaptation failures before publishing (T154116) (duration: 00m 50s)
  • 13:11 zfilipin@tin: Synchronized wmf-config/ProductionServices.php: SWAT: use the logstash LVS endpoint (T175242) (duration: 00m 51s)
  • 12:09 moritzm: removed obsolete packages from jessie-wikimedia/experimental
  • 12:06 mobrovac@tin: Finished deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 (duration: 08m 43s)
  • 11:57 mobrovac@tin: Started deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417
  • 11:43 elukey: drop table log.PageContentSaveComplete_5588433 from db1046,db1047,db1108,dbstore1002, archived on hdfs - T177101
  • 11:31 moritzm: remove obsolete packages from stretch-wikimedia/experimental, now empty
  • 11:29 mobrovac@tin: Finished deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 (duration: 08m 15s)
  • 11:26 elukey: drop log.MediaViewer_10867062_15423246 from db1047,db1108 since already archived in hdfs - T168303
  • 11:21 mobrovac@tin: Started deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417
  • 10:54 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1003.eqiad.wmnet
  • 10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1002.eqiad.wmnet
  • 10:53 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=logstash1001.eqiad.wmnet
  • 10:53 gehel: depooling logstash100[123] in preparation for decommission - T175830
  • 10:38 gehel: rolling restart of wdqs nodes for GC tuning - T175919
  • 10:13 marostegui: Deploy alter table on db2019 - T174569
  • 09:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T161088 (duration: 00m 50s)
  • 09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
  • 08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight after maintenance - T161088 (duration: 00m 50s)
  • 08:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after maintenance - T161088 (duration: 00m 50s)
  • 08:15 moritzm: restarting app server canaries to pick up curl security update
  • 08:04 moritzm: installing curl security updates
  • 07:59 marostegui: Stop mysql event scheduler on labsdb1001 - T179464
  • 06:54 marostegui: Stop s3 instance on db2092 to copy it to db2085 - T178359
  • 06:33 legoktm: done on mwdebug1002
  • 06:31 marostegui: Stop MySQL on db1091 and db1103 to migrate db1091 to file per table
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T161088 (duration: 01m 07s)
  • 06:27 marostegui: Deploy optimize table wikidatawiki.pagelinks on s5 master (db1063) - T174509
  • 06:02 legoktm: live-hacking mwdebug1002
  • 05:43 TimStarling: on puppetmaster2001: disk full due to logs, so I'm truncating /var/log/debug which is apparently copied to daemon.log anyway
  • 03:29 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 2 03:29:35 UTC 2017 (duration 7m 12s)
  • 03:22 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 37s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 06s)
  • 00:52 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/extension.json: SWAT followup: Update SearchSatisfaction eventlogging schema revision id (duration: 00m 50s)
  • 00:51 ebernhardson@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 50s)
  • 00:32 bblack: cp4022 - varnish backend restart, mailbox
  • 00:28 bblack: lvs1001-3: re-enable ethernet flowcontrol (short ethernet blip), with pybal stopped to failover to backups during...

2017-11-01

  • 23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 50s)
  • 23:35 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/WikimediaEvents: SWAT: Turn on Cirrus AB test for DBN group sizing (duration: 00m 51s)
  • 23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/ParserMigration/includes/ApiParserMigration.php: SWAT: API: Fix WikiPage namespace (duration: 00m 52s)
  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 49s)
  • 23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/Wikibase.hooks.php: SWAT: Allow turning Cirrus usage off from query T179428 (duration: 00m 51s)
  • 23:09 thcipriani@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Revert "Revert "Add negative weight to disambig entities"" T148411 (duration: 00m 51s)
  • 22:49 ejegg: disabled fundraising unsubscribe import processing
  • 22:33 Krinkle: group2(all-wikidata) wikis to wmf.5 from 24 hours ago seems to have caused a 60% drop in navigation timing metric report count (100/min => 40/min)
  • 22:19 eileen1: process-control updated to 9c8b4bb
  • 22:17 demon@tin: Synchronized wmf-config/CommonSettings.php: rel1_28 is dead (duration: 00m 50s)
  • 22:06 ejegg: re-enabled fundraising jobs
  • 21:48 ejegg: fundraising jobs disabled
  • 20:10 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.6
  • 20:05 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 19:57 krinkle@tin: Synchronized php-1.31.0-wmf.6/resources/src/startup.js: T178943 (duration: 00m 51s)
  • 19:53 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/CentralNotice: fix weird git rebasing issue (duration: 00m 53s)
  • 19:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.4 [keeping static files] (duration: 01m 52s)
  • 19:31 awight@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 50s)
  • 19:28 awight@tin: Synchronized php-1.31.0-wmf.6/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 52s)
  • 19:14 bblack: all hosts: manual cumin+sed removal of ethernet autoneg params from /e/n/i to match https://gerrit.wikimedia.org/r/#/c/387849/
  • 19:08 otto@tin: Finished deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610 (duration: 03m 45s)
  • 19:05 otto@tin: Started deploy [analytics/refinery@6d11d67]: Deploying refinery-source artifacts for 0.0.54 for JsonRefine job, T162610
  • 18:39 demon@tin: Synchronized wmf-config/: Dropping squid.php (hang on to your pants folks, this could be fun) (duration: 00m 51s)
  • 18:36 demon@tin: Synchronized wmf-config/CommonSettings.php: use reverse-proxy.php no more squid.php (duration: 00m 50s)
  • 18:35 demon@tin: Synchronized docroot/: dropping squid.php (duration: 00m 52s)
  • 18:13 demon@tin: Synchronized static/images/project-logos/sewikimedia.png: new logo (duration: 00m 50s)
  • 18:02 moritzm: installing openjpeg2 security updates
  • 17:39 demon@tin: Synchronized php-1.31.0-wmf.6/includes/specials/pagers/ContribsPager.php: fix missing page_is_new error (duration: 00m 51s)
  • 17:10 demon@tin: Synchronized wmf-config/: dropped squid-labs, no-op in prod (duration: 00m 52s)
  • 17:09 demon@tin: Synchronized docroot/noc/: dropped squid-labs.php (duration: 00m 51s)
  • 16:01 bblack: lvs1003 - puppet disabled, testing experimental ethtool ringbuffer change
  • 15:15 moritzm: repo reorg: moved ftpsync from thirdparty to main and docker-engine from thirdparty to thirdparty/k8s
  • 15:12 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production) (duration: 02m 25s)
  • 15:10 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores1002 (non-production)
  • 15:09 awight@tin: Finished deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production) (duration: 01m 57s)
  • 15:07 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
  • 15:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update CirrusSearch MLR model on enwiki (duration: 00m 51s)
  • 15:01 mobrovac: restbase: removing wikimedia-lvs-realserver from staging hosts T179494
  • 14:55 bblack: strongswan experiment done, cp* back to puppet-agent-enabled
  • 14:41 moritzm: installing poppler security updates
  • 14:09 bblack: cp*: disabling puppet to test strongswan change...
  • 13:49 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage on fawikiquote - T179442 (duration: 00m 50s)
  • 13:46 hashar@tin: Synchronized wmf-config/abusefilter.php: Enable blocking feature of abuse filter in fawikiquote - T178227 (duration: 00m 50s)
  • 13:45 moritzm: installing quagga security updates
  • 13:43 hashar@tin: Synchronized wmf-config/CommonSettings.php: Properly check for cluster existence prior setting TTM mirrors - T179270 (duration: 01m 05s)
  • 13:41 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2 (duration: 05m 50s)
  • 13:35 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency. Take 2
  • 13:33 ppchelko@tin: Finished deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency (duration: 03m 50s)
  • 13:29 ppchelko@tin: Started deploy [restbase/deploy@2321c4c]: Update hyperswitch dependency
  • 12:48 moritzm: installing libav security updates
  • 12:44 moritzm: installinng libdatetime-timezone-perl stable updates on Debian
  • 12:10 kartik@tin: Finished deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb (duration: 03m 07s)
  • 12:07 kartik@tin: Started deploy [cxserver/deploy@10651e2]: Update cxserver to 0227acb
  • 08:35 elukey: forced umount/mount for /mnt/hdfs on stat1005 (not working after repeated oom kill actions)
  • 03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 1 03:22:02 UTC 2017 (duration 7m 17s)
  • 03:14 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.6) (duration: 15m 21s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 46s)

2017-10-31

  • 23:56 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Adjust cirrussearch interleave configuration for AB test (duration: 00m 50s)
  • 23:48 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 50s)
  • 23:21 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 50s)
  • 23:19 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Setup CirrusSearch AB test on dbn group sizing (duration: 00m 53s)
  • 21:38 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
  • 21:37 awight@tin: Started deploy [ores/deploy@9f361d2]: revscoring 2 -> ores* (non-production)
  • 21:16 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/386553 https://gerrit.wikimedia.org/r/386710 (duration: 00m 52s)
  • 21:00 XioNoX: removing old AMS-IX IPv6 - T167840
  • 20:57 mepps: rollback payments-wiki from 5303a8e to b88b805
  • 20:54 mepps: updated payments-wiki from b88b805 to 5303a8e
  • 20:45 mepps: rolledback payments-wiki from 5303a8e to b88b805
  • 20:43 mepps: updated payments-wiki from b88b805 to 5303a8e
  • 19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.6
  • 18:53 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production) (duration: 10m 29s)
  • 18:42 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores* (non-production)
  • 18:38 awight@tin: Finished deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production) (duration: 02m 24s)
  • 18:36 awight@tin: Started deploy [ores/deploy@437fcf5]: revscoring 2 -> ores1002 (non-production)
  • 18:32 demon@tin: Synchronized php-1.31.0-wmf.6/extensions/CentralNotice: deployyyyy (duration: 00m 52s)
  • 18:27 ejegg: updated CiviCRM from a34a9c6 to 415fde1
  • 17:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata back to wmf.5
  • 16:50 demon@tin: Finished scap: bootstrap wmf.6 (duration: 47m 39s)
  • 16:41 ppchelko@tin: Finished deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias (duration: 09m 07s)
  • 16:32 ppchelko@tin: Started deploy [restbase/deploy@7631ea7]: Enable new parsoid storage on test wikipedias
  • 16:09 ariel@tin: Finished deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config (duration: 00m 02s)
  • 16:09 ariel@tin: Started deploy [dumps/dumps@fa0583b]: adapt dumpadmin script to use override section in config
  • 16:02 demon@tin: Started scap: bootstrap wmf.6
  • 15:43 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: all but wikidata.org -> wmf.5
  • 15:42 demon@tin: Synchronized php: Symlink bump (duration: 00m 48s)
  • 15:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 (duration: 03m 15s)
  • 14:18 bblack: authdns (baham, radon, eeden): upgrade base packages (curl, git, libc-dev, etc)
  • 14:13 bblack: lvsNNNN: upgrade base packages (curl, git, libc-dev, etc)
  • 14:11 bblack: cpNNNN: upgrade base packages (curl, git, libc-dev, etc)
  • 13:57 bblack: caches@eqiad - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:54 bblack: caches@esams - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:22 bblack: caches@codfw - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:18 zeljkof: EU SWAT finished
  • 13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.4/extensions/EventBus/EventBus.php: SWAT: Dont log full request to service in case of delivery error (T179280) (duration: 00m 51s)
  • 13:14 bblack: caches@ulsfo - upgrade nginx to 1.13.6-2+wmf1~jessie1
  • 13:05 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 02m 58s)
  • 13:02 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
  • 13:01 mobrovac@tin: Finished deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables (duration: 15m 19s)
  • 12:45 mobrovac@tin: Started deploy [restbase/deploy@c0f9dc4]: Cassandra 3 Parsoid stash tables
  • 12:39 bblack: eqiad primary lvses (1001-3): disable pause [LRO not possible] on all ethernet interfaces (under pybal stopped briefly)
  • 12:36 bblack: codfw primary lvses (2003-6): disable LRO,pause on all ethernet interfaces (under pybal stopped briefly)
  • 12:24 bblack: cp4025: restart varnish-be for mailbox lag
  • 12:23 bblack: cp4023: restart varnish-be for mailbox lag
  • 12:21 bblack: esams primary lvses (3001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
  • 12:19 bblack: ulsfo primary lvses (4001-2): disable LRO,pause on eth0 (under pybal stopped briefly)
  • 12:12 bblack: esams+ulsfo backup lvses (3003-4,4003-4): disable LRO,pause on eth0
  • 12:10 bblack: core DC backup lvses (1004-6,2004-6): disable LRO,pause on all ethernet interfaces
  • 11:05 oblivian@tin: Finished deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix (duration: 00m 04s)
  • 11:05 oblivian@tin: Started deploy [docker-pkg/deploy@20908e8]: Actually deploy the fix
  • 11:02 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter (duration: 00m 02s)
  • 11:02 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: Fix image_tag filter
  • 10:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T161088 (duration: 00m 48s)
  • 10:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
  • 09:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2084 as multi-instance core host T178553 T178359 (duration: 00m 49s)
  • 08:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T161088 (duration: 00m 50s)
  • 08:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T161088 (duration: 00m 50s)
  • 08:15 marostegui: Stop MySQL on db2086 to copy s5 to db2089 - T178359
  • 07:09 marostegui: Stop MySQL on db2087.s6 to copy its content to db2089 - T178359
  • 06:37 marostegui: Stop MySQL on db1103 and db1084 - T161088
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T161088 (duration: 00m 49s)
  • 06:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 (duration: 01m 12s)
  • 03:08 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 31 03:08:48 UTC 2017 (duration 7m 18s)
  • 03:01 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 10m 05s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 33s)

2017-10-30

  • 23:28 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Update cirrussearch MLR model for enwiki (duration: 00m 50s)
  • 23:22 ebernhardson@tin: Synchronized php-1.31.0-wmf.4/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 01s)
  • 23:19 ebernhardson@tin: Synchronized php-1.31.0-wmf.5/extensions/CirrusSearch/: Fix highlight tags inside section anchors of search results (duration: 01m 03s)
  • 22:03 eileen1: update process-control to revision is 2392ddf
  • 21:55 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336 (duration: 02m 03s)
  • 21:53 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Fun with deployments (non-production) T179336
  • 21:48 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 01m 42s)
  • 21:47 chasemp: restart nova-fullstack to force a run
  • 21:46 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 21:36 eileen1: update CiviCRM from 25c7a33 to a34a9c6
  • 21:02 urandom: T179083: Creating commons_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv.{meta,data} (30 sec delay)
  • 20:56 urandom: T179083: Creating commons_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH.{data,meta} (25 sec delay)
  • 20:46 urandom: T179083: Creating enwiki_T_parsoid_stash_section2ACMDK1DRzacK9nUB3.{data,meta} (15sec delay)
  • 20:36 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.data
  • 20:35 arlolra: Updated Parsoid to 292633c4 (T133334)
  • 20:34 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ.meta
  • 20:33 urandom: T179083: Creating enwiki_T_parsoid_stash_dataWH8IDUS9SGI6LPpsJsLOQ
  • 20:28 urandom: T179083: Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.data
  • 20:27 urandom: T179083: Creating table enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1.meta
  • 20:26 arlolra@tin: Finished deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4 (duration: 07m 33s)
  • 20:25 urandom: T179083: Creating keyspace enwiki_T_parsoid_stash_wikitextf0PBY8UXqY8UuiDv1
  • 20:20 urandom: T179083: Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.data
  • 20:18 arlolra@tin: Started deploy [parsoid/deploy@8a2c0ce]: Updating Parsoid to 292633c4
  • 20:14 urandom: T179083: Creating table enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5.meta
  • 20:09 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 14m 11s)
  • 20:08 urandom: T179083: Creating keyspace enwiki_T_parsoid_stash_htmlmXxc_uDhgnFQAdM8PPlH5
  • 19:55 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Appendix NS on Burmese Wiktionary T178545 (duration: 00m 55s)
  • 19:55 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 19:07 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 06m 16s)
  • 19:01 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 19:01 awight@tin: Finished deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production) (duration: 19m 53s)
  • 18:41 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 18:40 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 18:22 awight@tin: Started deploy [ores/deploy@a0f7d5c]: Smoke test for timeout bug on ores1002 (non-production)
  • 18:16 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
  • 18:15 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
  • 18:08 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add Timeless skin to wikitech T154371 (duration: 00m 51s)
  • 17:08 madhuvishy: Revert dns switchover for c1 shards to c3 post labsdb1001 reboot T168584
  • 17:07 gehel@tin: Finished deploy [wdqs/wdqs@fdb9100]: GUI update (duration: 02m 17s)
  • 17:05 gehel@tin: Started deploy [wdqs/wdqs@fdb9100]: GUI update
  • 16:45 oblivian@tin: Finished deploy [docker-pkg/deploy@576c80f]: New version with small fixes (duration: 00m 04s)
  • 16:44 oblivian@tin: Started deploy [docker-pkg/deploy@576c80f]: New version with small fixes
  • 15:20 XioNoX: re-enabling Zayo transit in eqiad
  • 14:36 marostegui: Reboot labsdb1001 - T168584
  • 14:30 marostegui: Stop replication and MySQL on labsdb1001 - T168584
  • 14:09 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107) (duration: 00m 50s)
  • 14:07 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Revert "UBN! disbale ores for wikidata" (T179107) (duration: 00m 49s)
  • 14:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Change the af.wiktionary logo, part II (T178824) (duration: 00m 50s)
  • 13:59 ladsgroup@tin: Synchronized static/images/project-logos: Change the af.wiktionary logo, part I (T178824) (duration: 00m 50s)
  • 13:40 madhuvishy: Switched dns over for c1 shards to c3 in preparation of labsdb1001 reboot (Merged https://gerrit.wikimedia.org/r/#/c/386660/, Ran puppet on labservice1001|2)
  • 13:21 marostegui: Set innodb_max_dirty_pages_pct = 10 on labsdb1001 so it powers off a bit faster - T168584
  • 13:19 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Autopatrolled User Rights & Ext:SandboxLink on hiwikiversity (T179251 T179252) (duration: 00m 50s)
  • 12:58 reedy@tin: Synchronized wmf-config/db-labs.php: beta labs stuffs (duration: 00m 50s)
  • 12:49 oblivian@tin: Finished deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version (duration: 00m 07s)
  • 12:49 oblivian@tin: Started deploy [docker-pkg/deploy@e9b9146]: Proper deploy of first version
  • 12:02 hoo: Manually started dumpwikidatajson on snapshot1007 (cron run failed tonight)
  • 12:02 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
  • 12:01 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
  • 12:00 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 19s)
  • 11:59 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
  • 11:57 oblivian@tin: Finished deploy [docker-pkg/deploy@549c157]: (no justification provided) (duration: 00m 11s)
  • 11:57 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: (no justification provided)
  • 11:55 oblivian@tin: Started deploy [docker-pkg/deploy@549c157]: Test deploy for virtualenv creation
  • 11:48 hoo: Ran rebuildPropertyInfo for wikidatawiki and manually cleared the property cache (mc)
  • 11:44 hoo@tin: Synchronized wmf-config/Wikibase.php: Re-add property for RDF mapping of external identifiers for Wikidata (T179156, T178180) (duration: 00m 49s)
  • 11:33 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Re-enable constraints check with SPARQL (T179156) (duration: 00m 50s)
  • 11:24 marostegui: Compress table cebwiki.externallinks on db2092 - T178359
  • 11:15 hoo@tin: rebuilt wikiversions.php and synchronized wikiversions files: Wikidatawiki back to wmf.5 (T179156)
  • 10:58 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion (duration: 08m 33s)
  • 10:52 akosiaris: poweroff copper T176957
  • 10:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch, take #gazillion
  • 10:31 marostegui: Stop MySQL on db2039 to copy its data to db2087.s6 - T178359
  • 10:27 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 04m 41s)
  • 10:22 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
  • 10:03 mobrovac@tin: Finished deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch (duration: 12m 12s)
  • 09:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1097 original weight - T161088 (duration: 00m 52s)
  • 09:50 mobrovac@tin: Started deploy [restbase/deploy@2b5889b]: Double-process all summaries and include the parsoid no-op switch
  • 09:22 marostegui: Stop replication in sync on db2039 and db2046 to reimport tables
  • 09:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1097 api traffic - T161088 (duration: 00m 50s)
  • 09:08 marostegui: Drop wb_entity_per_page page from s3 and s5 - T177601
  • 08:51 ema: cp4022: restart varnish-be for mbox lag
  • 08:42 elukey: raised priority of refreshlink and htmlcacheupdate job execution on jobrunners (https://gerrit.wikimedia.org/r/#/c/386636/) - T173710
  • 08:39 marostegui: Stop replication on db2039 to reimport and compress tables
  • 08:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 with low weight - T161088 (duration: 00m 50s)
  • 08:33 moritzm: installing wget security updates
  • 08:32 gehel: rolling restart of wdqs for config reload - T175919
  • 08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 (duration: 00m 50s)
  • 08:28 ariel@tin: Finished deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script (duration: 00m 02s)
  • 08:28 ariel@tin: Started deploy [dumps/dumps@c204c72]: fix args initialization issue in getconfigvals script
  • 08:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 - T178359 (duration: 00m 50s)
  • 08:11 marostegui: Stop MySQL on db2086 to transfer s7 to db2087 - T178359
  • 06:51 marostegui: Stop MySQL on db1097 to clone db1103 - T161088
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 50s)
  • 06:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1103 from s3 to s4 - T161088 (duration: 00m 49s)
  • 06:40 marostegui: Stop MySQL on db1103 to reclone it - T161088
  • 06:24 marostegui: Optimize dewiki.pagelinks and dewiki.templatelinks on db1063 (s5 master) - T174509
  • 06:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 - T178359 (duration: 00m 51s)
  • 06:08 marostegui: Stop MySQL on db2040 to populate s7 on db2086 - T178359
  • 05:58 marostegui: Deploy alter table on db1075 (s3 primary master) - T174509
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 30 03:11:37 UTC 2017 (duration 7m 11s)
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 11m 38s)
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 11m 18s)

2017-10-29

  • 23:49 ema: powercycle cp4024
  • 22:31 ariel@tin: Finished deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides (duration: 00m 02s)
  • 22:31 ariel@tin: Started deploy [dumps/dumps@2aa2275]: fix keep setting to work with overrides
  • 17:55 ariel@tin: Finished deploy [dumps/dumps@d8978ce]: add overrides section processing to config file (duration: 00m 04s)
  • 17:55 ariel@tin: Started deploy [dumps/dumps@d8978ce]: add overrides section processing to config file
  • 17:23 ariel@tin: Finished deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup (duration: 00m 02s)
  • 17:23 ariel@tin: Started deploy [dumps/dumps@d426cf7]: batch 7z jobs, multistream job fixup
  • 12:54 ema: cp4026: restart varnish-be for mbox lag

2017-10-28

  • 21:03 bblack: cp1067 (current target cache): disabling the relatively-new VCL that sets do_stream=false if !CL on applayer fetches...
  • 19:39 hoo@tin: Synchronized wmf-config/CommonSettings.php: Half the Flow -> Parsoid timeout (100s -> 50s) (T179156) (duration: 00m 51s)
  • 19:39 bblack: backend restart on cp1065
  • 18:39 bblack: restarting varnish backend on cp1053 to move the lag/503 issues to another box and buy more time to debug
  • 18:28 bblack: cp4025 - restart backend for mailbox lag (upload@ulsfo, unrelated to text-cluster issues)
  • 18:21 bblack: cp1053 - manual VCL change, backends appservers+api_appservers, reduce connect/firstbyte/betweenbytes timeoues from 5/180/60 to 3/20/10
  • 16:51 elukey: restart varnish backend on cp1055 - mailbox lag + T179156
  • 12:14 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1313.eqiad.wmnet
  • 12:10 elukey: manually killed (SIGTERM) hhvm on mw1313 - high load, hhvm-dump-debug not responsive
  • 12:01 elukey@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1313.eqiad.wmnet
  • 11:53 elukey: restart hhvm on mw1285 - hhvm-dump-debug in /tmp/hhvm.17700.bt
  • 11:24 hoo@tin: Synchronized wmf-config/Wikibase-labs.php: Consistency sync (duration: 00m 50s)
  • 10:52 volans: restarted pdfrender on scb1001, was stuck since 2d with AssertionError: display is not set!

2017-10-27

  • 20:54 MaxSem: running migratePreferences.php on group2 wikis
  • 19:09 hoo: Ran scap pull on mwdebug1001
  • 18:20 awight@tin: Finished deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production) (duration: 02m 17s)
  • 18:18 awight@tin: Started deploy [ores/deploy@185170f]: Test pip-9 scap trick on ores1002 (non-production)
  • 17:54 hoo: Taking mwdebug1001 to do tests regarding T179156
  • 16:38 gehel: re-enabling wdqs-updater
  • 16:16 bblack: cp1054 varnish backend restarted (was 503s / bad-conns target of ongoing issues)
  • 16:16 gehel: wdqs updater is now stopped for real
  • 16:10 XioNoX: deactivating BGP sessions to Zayo in eqiad (flapping)
  • 15:58 gehel: disabling wdqs updater on all nodes
  • 15:50 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Disable constraints check with SPARQL for now (T179156) (duration: 00m 50s)
  • 15:48 marostegui: Compress InnoDB on db2039 (s6) - T178359
  • 15:46 bblack: restart varnish-backend on cp4022 (upload@ulsfo) - mailbox
  • 14:49 bblack: turn on cp4024 port on asw-ulsfo
  • 13:52 bblack: reboot cp4021 to clean up oom messes
  • 13:49 bblack: restarting nginx on cp4021, without NUMA memory constraints
  • 12:10 marostegui: Optimize commonswiki.templatelinks on dbstore1001 - T162789
  • 12:03 elukey: execute systemctl reset-failed kafka-mirror-main-eqiad_to_jumbo-eqiad.service on kafka-jumbo hosts (old unit not deployed anymore)
  • 11:41 mutante: gerrit back - maintenance over
  • 11:39 mutante: gerrit restart to apply gerrit:386793 is imminent
  • 11:36 ema: cp4023: varnish-backend-restart for lag
  • 11:06 mobrovac@tin: Finished deploy [citoid/deploy@ff63420]: Update dependencies (duration: 03m 18s)
  • 11:04 marostegui: Optimize cebwiki.templatelinks on db1103 - T174509
  • 11:03 mobrovac@tin: Started deploy [citoid/deploy@ff63420]: Update dependencies
  • 08:56 marostegui: Optimize table commonswiki.recentchanges on dbstore1001 - T162789
  • 08:11 marostegui: Drop redundant indexes from pagelinks and templatelinks on s3 wikis only for dbstore2001 and dbstore2002 - T174509
  • 05:59 marostegui: Stop MySQL on db2084 to copy s5 to db2086 - T178359
  • 05:55 marostegui: dbstore1001: convert itwiki.pagelinks to TokuDB - T162789
  • 05:53 mutante: uploaded parsoid_0.8.0 to releases.wikimedia.org (for Subbu - T179134)
  • 05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174509 (duration: 00m 50s)
  • 03:02 bblack: cp1067, cp4026 - backend restarts, mailbox lag

2017-10-26

  • 22:47 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wikidata to wmf.4
  • 22:39 bblack: restarting varnish-backend on cp1053 (mailbox lag from ongoing issues elsewhere?)
  • 21:40 bblack: raising backend max_connections for api.svc.eqiad.wmnet + appservers.svc.eqiad.wmnet from 1K to 10K on cp1053.eqiad.wmnet (current funnel for the bulk of the 503s)
  • 21:32 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s)
  • 21:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s)
  • 21:00 hoo: Fully revert all changes related to T178180
  • 20:59 hoo@tin: Synchronized wmf-config/Wikibase.php: Revert "Add property for RDF mapping of external identifiers for Wikidata" (T178180) (duration: 00m 50s)
  • 20:02 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s)
  • 20:00 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s)
  • 19:41 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 07m 25s)
  • 19:36 aaron@tin: Started restart [jobrunner/jobrunner@a20d043]: (no justification provided)
  • 19:34 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
  • 19:33 awight@tin: Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 00m 10s)
  • 19:33 awight@tin: Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107
  • 19:27 demon@tin: Synchronized php-1.31.0-wmf.5/includes/libs/filebackend/SwiftFileBackend.php: Better error grouping (duration: 00m 50s)
  • 19:14 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 00m 49s)
  • 19:12 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 50s)
  • 18:52 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 49s)
  • 18:51 ladsgroup@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 01m 04s)
  • 18:41 bblack: eqiad cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 18:30 urandom: Dropping and recreating keyspace enwiki_T_page__summary (restbase-ng cluster)
  • 18:27 bblack: esams cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 18:26 urandom: T179105: Altering "enwiki_T_page__summary".data to use LZ4Compressor
  • 17:59 bblack: codfw cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
  • 17:49 bblack: ulsfo cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause
  • 17:47 awight@tin: Finished deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) (duration: 01m 45s)
  • 17:45 awight@tin: Started deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production)
  • 17:36 arlolra: Updated Parsoid to 8e99708a (T176728)
  • 17:30 arlolra@tin: Finished deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a (duration: 08m 40s)
  • 17:21 arlolra@tin: Started deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a
  • 17:17 awight@tin: Finished deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 (duration: 00m 32s)
  • 17:16 awight@tin: Started deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441
  • 17:14 awight@tin: Finished deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 (duration: 03m 20s)
  • 17:11 awight@tin: Started deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441
  • 17:09 marostegui: Optimize ruwiki.recentchanges on dbstore1001 - T162789
  • 17:08 dcausse: [logstash] reimporting today logs
  • 17:08 dcausse: [logstash] deleting today logs to reapply mapping template
  • 16:58 ottomata: now mirroring main Kafka cluster topics to jumbo Kafka cluster, with MirrorMaker instances running on analytics-eqiad broker nodes. https://phabricator.wikimedia.org/T177216
  • 16:44 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441 (duration: 01m 02s)
  • 16:43 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441
  • 16:33 awight@tin: Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441 (duration: 15m 33s)
  • 16:21 urandom: restarted cassandra, restbase2001-b
  • 16:18 awight@tin: Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441
  • 16:15 urandom: truncating cassandra hints in restbase-ng cluster
  • 16:12 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP (duration: 07m 57s)
  • 16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
  • 16:04 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP
  • 15:50 mobrovac@tin: Finished deploy [restbase/deploy@522321a]: Double-process all summaries (duration: 09m 09s)
  • 15:45 urandom: restarting cassandra, restbase2001-b
  • 15:41 andrewbogott: upgrading dnsmasq on labnet1001 and labnet1002
  • 15:41 mobrovac@tin: Started deploy [restbase/deploy@522321a]: Double-process all summaries
  • 15:36 dcausse: reindexing logs in logstash
  • 15:34 dcausse: deleting and reimporting logs in logstash (today data, might lose some logs in the process)
  • 14:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101 - T178383 (duration: 00m 49s)
  • 14:04 zeljkof: EU SWAT finished
  • 14:03 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikibase: SWAT: Make search for titles be always uppercase (T179045) (duration: 01m 36s)
  • 13:52 zfilipin@tin: Synchronized php-1.31.0-wmf.5/extensions/Wikidata: SWAT: Make search for titles be always uppercase (T179045) (duration: 02m 10s)
  • 13:46 Amir1: ladsgroup@terbium:~$ mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildPropertyInfo.php --wiki=wikidatawiki --rebuild-all --force (T178180)
  • 13:41 godog: delete old CF cassandra metrics - T173436
  • 13:32 godog: force delete old graphite cassandra metrics - T179057
  • 13:28 moritzm: installing icu security update on trusty
  • 13:28 zfilipin@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add property for RDF mapping of external identifiers for Wikidata (T178180) (duration: 00m 50s)
  • 13:21 marostegui: Compress InnoDB on db2040 - T178359
  • 13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable signature button in Projekt namespace on sewikimedia (T175363) (duration: 00m 50s)
  • 13:13 zfilipin@tin: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Fix interwiki sort order for Northern Sami (T178965) (duration: 00m 50s)
  • 12:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Reorganize future rc multi-instance hosts - T178359 (duration: 00m 50s)
  • 12:30 bblack: powercycle cp4024 (depooled)
  • 12:29 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet
  • 10:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clean up some comments in s3 (duration: 00m 50s)
  • 09:01 moritzm: upgrade script runners to wikidiff2 1.5.1
  • 08:58 ema: cache_text/upload varnish-be: set rutime parameter timeout_idle to 120s
  • 08:34 moritzm: uploaded php-wikidiff2/hhvm-wikidiff2 1.5.1 for stretch-wikimedia
  • 08:22 godog: add 100G to graphite1003's /var/lib/carbon
  • 08:14 moritzm: upgrade deployment servers to wikidiff2 1.5.1
  • 07:57 marostegui: Drop databases in s1 and s2 from db1047 and unconfigure replication - T177405
  • 07:43 marostegui: Stop MySQL on db1047 to copy data over db1108 - T177405 T156844
  • 07:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 00m 50s)
  • 07:12 marostegui: Stop replication in sync on db1103 and db1077 to fix data drifts - T164488
  • 07:10 moritzm: upgrade mw1319-mw1328 to wikidiff2 1.5.1
  • 06:57 marostegui: Stop MySQL on db2088 and db2084 to copy s2 and s4 to db2091 - T178359
  • 06:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 50s)
  • 06:55 marostegui: Optimize pagelinks and templatelinks on db1095 s1 - T174509
  • 06:51 marostegui: Optimize recentchanges on s4 and s6 on labsdb1010 - T177772
  • 06:15 moritzm: upgrade mw1299-mw1311 to wikidiff2 1.5.1
  • 05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 - T174509 (duration: 00m 50s)
  • 05:14 marostegui: Optimize recentchanges on s4 and s6 on labsdb1009 - T177772
  • 05:12 marostegui: Optimize pagelinks and templatelinks on db1060 - T174509
  • 03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 26 03:01:19 UTC 2017 (duration 6m 48s)
  • 02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 09m 30s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 10m 34s)

2017-10-25

  • 23:48 legoktm@tin: Synchronized php-1.31.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: MWInternalLinkContextItem: increase specificity to override OOUI changes - T178933 (duration: 00m 50s)
  • 23:39 legoktm@tin: Synchronized wmf-config/abusefilter.php: Change threshold for slow AbuseFilter logging to 800ms - T179039 (duration: 00m 50s)
  • 22:42 MaxSem: running migratePreferences.php on group1 wikis
  • 22:01 mobrovac@tin: Finished deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2 (duration: 09m 09s)
  • 21:52 mobrovac@tin: Started deploy [restbase/deploy@860cbfe]: Revert double-processing of all summaries to all except WPs, take #2
  • 21:48 mobrovac@tin: Started deploy [restbase/deploy@60ce036]: Revert double-processing of all summaries to all except WPs
  • 21:33 bblack: experimental - disabled LRO on lvs4001+lvs4002
  • 20:34 hasharAway: restarted zuul to flush a long queue in experimental pipeline (no other changes enqueued)
  • 20:21 demon@tin: Synchronized php-1.31.0-wmf.5/extensions/ProofreadPage/: unbreak multiline text input (duration: 00m 53s)
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.5
  • 18:45 bblack: s/1\.13\.16/1.13.6/
  • 18:45 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable fine grained usage tracking on statement usage wikis T172914 (duration: 00m 50s)
  • 18:44 bblack: reprepro: uploaded nginx-1.13.16-2+wmf1 packages to stretch and jessie
  • 18:36 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Make disabled usage aspect use the new config T172914 (duration: 00m 50s)
  • 18:33 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/Wikidata: Allow specifying a replacement usage type in disabledUsageAspects T178153 (duration: 02m 10s)
  • 18:25 niharika29@tin: Synchronized wmf-config/Wikibase.php: Make using CirrusSearch engine default for wbsearchentities T175741 (duration: 00m 48s)
  • 18:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add AbuseFilterSlow channel to monolog T178853 (duration: 00m 50s)
  • 18:20 niharika29@tin: Synchronized php-1.31.0-wmf.5/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 50s)
  • 18:19 niharika29@tin: Synchronized php-1.31.0-wmf.4/extensions/ORES/: Update ApiHooks to conform with v3 responses T178962 (duration: 00m 51s)
  • 18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Special:EmailUser User Prohibit on All Wikis [mediawiki-config] - https://gerrit.wikimedia.org/r/386231 (duration: 00m 51s)
  • 17:17 moritzm: upgrade mw2224-mw2242 to wikidiff2 1.5.1
  • 16:59 moritzm: upgrade mw2190-mw2199 to wikidiff2 1.5.1
  • 16:38 no_justification: gerrit: purging last remnants of debian package and running puppet. service may briefly restart
  • 16:05 moritzm: upgrade mw2180-mw2189 to wikidiff2 1.5.1
  • 16:03 moritzm: copied arcconf, hp-health, hpacucli, hpssa, hpssacli, hpssaducli, lsiutil, megacli, megaclisas-status to stretch-wikimedia/thirdparty/hwraid (thirdparty will be dropped at a later point)
  • 15:39 marostegui: Optimize pagelinks and templatelinks on labsdb1011 - T174509
  • 15:31 marostegui: Power off db1101 for HW maintenance - T178383
  • 15:21 godog: bounce rsyslog on wezen
  • 15:09 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T177836: Deploy Compact Language Links on the German Wikipedia (duration: 00m 49s)
  • 14:52 gehel: applying new template to elasticsearch / logstash - T178530
  • 14:51 ebernhardson: update logstash template on logstash elsaticsearch cluster for T178530
  • 14:47 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T166759: Set $wgAutoloadAttemptLowercase to false (duration: 00m 50s)
  • 14:37 mattflaschen@tin: Synchronized php-1.31.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
  • 14:36 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T177956: [cirrus] Turn off recall A/B test on enwiki (duration: 00m 49s)
  • 14:25 moritzm: moved jmxtrans and prometheus-jmx-exporter from thirdparty to main as part of repo reorg for stretch-wikimedia
  • 14:11 matt_flaschen: 'Deployed T178933: MWInternalLinkContextItem: increase specificity to override OOUI changes'
  • 14:10 mattflaschen@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/contextitems/ve.ui.MWInternalLinkContextItem.css: (no justification provided) (duration: 00m 50s)
  • 13:36 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings.php: T168886: Updating wikis with consolidate editing feedback (duration: 00m 51s)
  • 13:30 godog: deploy thumbor 1.7 - T178974
  • 13:30 elukey: restart yarn nodemanager and hdfs datanode on analytics1030 to apply new JVM settings - T178876
  • 13:15 chasemp: merge arturo's key as part of ops
  • 13:04 matt_flaschen: About to run ULSCompactLinksDisablePref.php: Setting user preferences for Compact Language Links on dewiki, before removing Beta Feature and changing to regular preference. T177836
  • 12:36 moritzm: upgrade mw2163-mw2179 to wikidiff2 1.5.1
  • 12:17 mobrovac@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Activate warning logging for JobExecutor (duration: 00m 50s)
  • 12:16 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Activate warning logging for JobExecutor (duration: 00m 50s)
  • 12:12 moritzm: upgrade mw1266-mw1275 to wikidiff2 1.5.1
  • 11:30 moritzm: upgrade mw1280-mw1290, mw1312-mw1318 to wikidiff2 1.5.1
  • 11:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s)
  • 10:49 mobrovac@tin: Finished scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ (duration: 04m 43s)
  • 10:45 mobrovac@tin: Started scap: wmf-config/jobqueue-labs.php beta-only change for CP4JQ
  • 10:41 mobrovac@tin: Finished deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100 (duration: 09m 41s)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T164488 (duration: 00m 50s)
  • 10:32 moritzm: upgrade remaining video scalers in eqiad to wikidiff2 1.5.1
  • 10:31 mobrovac@tin: Started deploy [restbase/deploy@2ae5b0c]: Remove /title/{title}/ and double-process all summaries - T158100
  • 09:31 moritzm: upgrade mw1221-mw1235 to wikidiff2 (API servers)
  • 09:07 gehel: rolling restart of all wdqs nodes for GC config change - T175919
  • 08:12 moritzm: upgrade mw1238-mw1258 to wikidiff2
  • 08:02 _joe_: decommissioning mw1168/69 as videoscalers, stopped jobrunner/jobchron, will stop hhvm/nginx/apache once current processing is done
  • 07:42 gehel@tin: Finished deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842 (duration: 01m 44s)
  • 07:40 gehel@tin: Started deploy [wdqs/wdqs@0bb2b5c]: wdqs-updater upgrade for jolokia - T167842
  • 07:05 marostegui: Optimize pagelinks and templatelinks on db1021 - T174509
  • 06:15 marostegui: Stop MySQL on db2038 to copy its data to db2084 - T178359
  • 06:08 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 - T178359 (duration: 00m 50s)
  • 06:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065 - T174509 (duration: 00m 50s)
  • 05:44 marostegui: Stop replication in sync on db1103 and db1077 to checksum data - T164488
  • 05:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T164488 (duration: 01m 00s)
  • 03:22 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 25 03:22:51 UTC 2017 (duration 7m 20s)
  • 03:15 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.5) (duration: 16m 46s)
  • 02:35 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 59s)

2017-10-24

  • 22:54 ejegg: updated civicrm from from 0546446 to 25c7a33
  • 21:24 ejegg: updated civicrm from 18edce1 to 0546446
  • 21:21 mepps: updated payments-wiki from 1011434 to b88b805
  • 20:47 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: (no justification provided) (duration: 00m 13s)
  • 20:47 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: (no justification provided)
  • 20:29 mutante: restarting gerrit to make gerrit-ssh listen on all IPs again
  • 19:49 arlolra@tin: Finished deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts (duration: 00m 35s)
  • 19:48 arlolra@tin: Started deploy [parsoid/deploy@UNKNOWN]: Enabling logging for skwiki posts
  • 19:46 mutante: restarting gerrit on cobalt to apply change 354074
  • 19:13 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.5
  • 18:57 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180 (duration: 10m 13s)
  • 18:47 awight@tin: Started deploy [ores/deploy@fb55ab8]: Rollback ORES to Revscoring 1.0, T175180
  • 18:47 ejegg: updated civicrm from b9d2067 to 18edce1
  • 18:26 awight@tin: Finished deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2) (duration: 10m 03s)
  • 18:16 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180 (take 2)
  • 18:05 arlolra: Updated Parsoid to a3be9cfc (T96923, T177784, T176568, T25467)
  • 17:57 arlolra@tin: Finished deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc (duration: 09m 16s)
  • 17:48 arlolra@tin: Started deploy [parsoid/deploy@af55c4b]: Updating Parsoid to a3be9cfc
  • 17:28 demon@tin: Finished scap: bootstrap wmf.5 (duration: 48m 34s)
  • 17:18 urandom: T178845: Rolling Cassandra restart, eqiad
  • 17:07 awight@tin: Started deploy [ores/deploy@fb55ab8]: Update ORES to revscoring 2.x, T175180
  • 16:50 herron: upgrading puppet packages on codfw puppetmaster T177254
  • 16:39 demon@tin: Started scap: bootstrap wmf.5
  • 16:10 demon@tin: Pruned MediaWiki: 1.31.0-wmf.3 [keeping static files] (duration: 01m 20s)
  • 15:58 elukey: drop MediaViewer_10867062_15423246 and MobileWebUIClickTracking_10742159_15423246 from the log database on db1046 (archived on hadoop) - T168303
  • 15:53 urandom: T178845: Rolling Cassandra restart, codfw, row b
  • 15:48 elukey: drop table Edit_13457736_15423246 from the log database (Eventlogging) on db104[6,7], dbstore1002
  • 15:39 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 to mw[1308-1311] - T136094
  • 15:09 gehel: restarting blazegraph on all wdqs nodes for GC config change - T175919
  • 14:48 _joe_: also stopping HHVM, nginx, apache2 , and disabling all the services
  • 14:46 _joe_: stopping the jobrunner service on mw1161-67
  • 14:39 marostegui: Optimize pagelinks templatelinks and recentchanges on db1030 - T174509 https://phabricator.wikimedia.org/T177772
  • 14:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065 - T174509 (duration: 00m 45s)
  • 14:33 marostegui: Optimize ores_classification on enwiki db1065 - T159753
  • 14:32 marostegui: Optimize pagelinks and templatelinks on db1065 - T174509
  • 14:24 gehel: restarting blazegraph on wdqs2001 for GC config change - T175919
  • 14:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1106 weight (duration: 00m 45s)
  • 13:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 - T178359 (duration: 00m 45s)
  • 13:41 marostegui: Stop MySQL on db2035 to copy its data to db2088 - T178359
  • 13:39 zeljkof: EU SWAT finished
  • 13:39 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add $wgNamespaceRobotPolicies Config for hewikisource (T178775) (duration: 00m 46s)
  • 10:50 dcausse: cirrus/elasticsearch: reindexing of 167 small wikis done (T177871)
  • 09:22 marostegui: Stop s1 on db2092 to copy its data to db2088 - T178359
  • 08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T178460 (duration: 00m 45s)
  • 07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T178460 (duration: 00m 45s)
  • 07:46 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
  • 07:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
  • 07:28 marostegui: Stop replication in sync on db1078 and db1103 to fix data drifts - https://phabricator.wikimedia.org/T164488
  • 06:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
  • 06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T178460 (duration: 00m 47s)
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 24 02:32:25 UTC 2017 (duration 6m 33s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 13s)

2017-10-23

  • 23:09 demon@tin: Synchronized wmf-config/throttle.php: Working Class Movement Library (Salford) throttle rule (duration: 00m 46s)
  • 22:59 bblack: cp4024: varnish-backend-restart for lag
  • 21:40 XioNoX: increasing MTU on Zayo transit interfaces
  • 20:34 arlolra: Updated Parsoid to 1cefef12 (T178217)
  • 20:25 arlolra@tin: Finished deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12 (duration: 11m 14s)
  • 20:14 arlolra@tin: Started deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12
  • 19:37 bsitzmann@tin: Finished deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828) (duration: 06m 59s)
  • 19:30 bsitzmann@tin: Started deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828)
  • 19:00 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 02m 34s)
  • 18:58 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
  • 18:58 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 05m 07s)
  • 18:52 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix
  • 18:52 joal@tin: Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests (duration: 321m 51s)
  • 17:56 gehel@tin: Finished deploy [wdqs/wdqs@b84592f]: WDQS GUI update (duration: 01m 58s)
  • 17:54 gehel@tin: Started deploy [wdqs/wdqs@b84592f]: WDQS GUI update
  • 17:48 XenoRyet: updated smashpig from d056a13 to 5262d53
  • 17:32 demon@tin: Synchronized scap/plugins/prep.py: Fixes and improvements of various sorts (duration: 00m 45s)
  • 17:26 gehel@tin: Finished deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates (duration: 02m 07s)
  • 17:24 gehel@tin: Started deploy [wdqs/wdqs@4626acb]: deploying latest WDQS version, blazegraph + GUI updates
  • 16:53 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 (duration: 03m 09s)
  • 15:59 elukey: forced BBU learn cycle on db1046 - T166141
  • 15:00 herron: wiped resolver cache of records puppet.cofdw.wmnet and puppet.ulsfo.wmnet (T177254)
  • 14:46 herron: depooling codfw puppetmaster (via dns)
  • 14:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
  • 14:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T178359 (duration: 00m 46s)
  • 13:48 marostegui: Stop mysql and poweroff db1082 for HW maintenance - T178460
  • 13:30 joal@tin: Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests
  • 13:19 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add aleph500.biblacad.ro to $wgCopyUploadsDomains - T178753 (duration: 00m 47s)
  • 13:19 gehel: rolling restart of wdqs for GC tuning - T175919
  • 13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 47s)
  • 13:08 zeljkof: EU SWAT finished, no deploys
  • 13:00 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4018.ulsfo.wmnet
  • 12:59 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4032.ulsfo.wmnet
  • 12:53 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4017.ulsfo.wmnet
  • 12:47 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4031.ulsfo.wmnet
  • 12:46 moritzm: upgrading hhvm-wikdiff2 on remaining job runners in codfw
  • 12:39 moritzm: upgrading hhvm-wikdiff2 on remaining video scalers in codfw
  • 12:32 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4010.ulsfo.wmnet
  • 12:29 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Remove hooserv.net from wgCopyUploadsDomains (duration: 00m 46s)
  • 12:26 moritzm: upgrading hhvm-wikdiff2 on mw2153-mw2162 (job runners)
  • 12:21 moritzm: upgrading hhvm-wikdiff2 on mw2254-mw2258 (app servers)
  • 12:20 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4030.ulsfo.wmnet
  • 12:15 bblack@neodymium: conftool action : set/pooled=no; selector: dc=ulsfo,cluster=cache_text,name=cp4009.ulsfo.wmnet
  • 12:12 bblack@neodymium: conftool action : set/pooled=yes; selector: dc=ulsfo,cluster=cache_text,name=cp4029.ulsfo.wmnet
  • 12:03 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on further wikis (T178515) (duration: 00m 47s)
  • 11:11 moritzm: upgrading hhvm-wikdiff2 on mw2097-mw2114
  • 11:10 marostegui: Compress innodb on db2038 and db2084 - T178359
  • 10:56 moritzm: upgrading hhvm-wikdiff2 on remaining API servers in codfw
  • 10:46 moritzm: upgrading hhvm-wikdiff2 on mw1161-mw1167 (job runners)
  • 10:28 moritzm: upgrading hhvm-wikdiff2 on mw1180-mw1188 (app servers)
  • 10:28 marostegui: Remove /srv from db1015 as it has been stopped for weeks now and will be decommissioned (and it is alerting low on disk space) - T173570
  • 10:09 dcausse: elasticsearch/cirrus reindexing 167 wikis from terbium (T177871)
  • 10:08 gehel: restart tilerator / karthotherian for config change on all maps clusters - T160215
  • 10:01 ema: restart varnish-be on cp4021 (mbox lag)
  • 09:58 moritzm: upgrading hhvm-wikidiff2 on mw1293-mw1298 (scalers)
  • 09:09 moritzm: upgrading hhvm-wikidiff2 on mw1189-mw1208 (API servers)
  • 08:09 marostegui: Rename wb_entity_per_page table on s5 db1092 and s3 db1077 - T177601
  • 08:07 moritzm: upgrading hhvm-wikdiff2 on mw1209-mw1220 (app servers)
  • 07:23 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 and db2038 - T178359 (duration: 00m 46s)
  • 06:56 moritzm: installing expat security updates on trusty
  • 06:51 marostegui: Stop replication in sync on db1078 and db1103 to checksum data - T164488
  • 06:50 moritzm: installing poppler security updates on trusty
  • 06:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 47s)
  • 06:30 marostegui: Stop replication in sync on db1103 and db2018 to fix data drifts - T164488
  • 06:09 marostegui: Stop replication on db2092 to re-import and compress linter and watchlist table
  • 02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 23 02:41:04 UTC 2017 (duration 6m 55s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 45s)

2017-10-21

  • 08:16 hashar: Mass code-review+2 changes made by LibraryUpdater that already had Code-Review:+2 and NOT verified=-1 ( ping legoktm )
  • 08:08 hashar: Stopping Zuul to flush its queue

2017-10-20

  • 23:46 demon@tin: Finished deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin (duration: 00m 09s)
  • 23:46 demon@tin: Started deploy [gerrit/gerrit@21b7332]: pushing deleteproject plugin
  • 19:19 hasharAway: restarting Jenkins - T178608
  • 18:52 bblack: vhtcpd upgrade + queue delay puppetization deploy ( https://gerrit.wikimedia.org/r/385415 ) done on cp* fleet - T133821
  • 18:28 bblack: upgrading vhtcpd to 0.1.1-1 on the cp* fleet
  • 17:38 mutante: removing Apache and HHVM Ganglia stats from all appservers, part of retiring Ganglia (T177225) - some transient puppet issues while purging package and remnants - use grafana dashboard at https://grafana.wikimedia.org/dashboard/db/prometheus-apache-hhvm-dc-stats
  • 16:39 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/TwoColConflict: 5c3224980fe (duration: 00m 46s)
  • 16:37 aaron@tin: Synchronized php-1.31.0-wmf.4/extensions/OATHAuth: c16a94e17b9981 (duration: 00m 47s)
  • 16:05 krinkle@tin: Synchronized php-1.31.0-wmf.4/extensions/WikimediaMaintenance/getJobQueueLengths.php: I8737795e6 (duration: 00m 47s)
  • 16:05 XioNoX: removing "policy-options policy-statement LVS_import term secondary" on cr routers
  • 14:39 cmjohnson1: updating power firmware analytics1037
  • 10:14 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2150/mw2151/mw2244/mw2245
  • 09:17 moritzm: upgrading hhvm-wikidiff2 to 1.5.1 on mw2200-mw2223
  • 08:05 Amir1: mwscript createAndPromote.php --wiki=techconductwiki --sysop --bureaucrat 'Ladsgroup' --force
  • 07:39 jynus: stopping db2033, too
  • 07:36 jynus: stopping dbstore1002 and dbstore2002 x1 replication in sync
  • 07:18 _joe_: uploading updated versions of docker base images (wikimedia-jessie, wikimedia-stretch) to docker-registry.wikimedia.org
  • 06:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s)
  • 06:17 marostegui: Stop replication in sync on db1103 and db1078 - T164488
  • 06:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s)
  • 06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 46s)
  • 05:52 marostegui: Stop replication in sync on db1103 and db1072 - T164488
  • 05:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1051 - T174509 (duration: 00m 47s)
  • 02:19 bblack: upgrading vhtcpd to 0.1.1-1 on: cp1008, cp3030, cp3034, cp4029-32
  • 02:17 bblack: uploaded vhtcpd-0.1.1-1 to reprepro

2017-10-19

  • 22:21 addshore: stop running extra change dispatchers on terbium for wikidatawiki
  • 21:58 addshore: running extra change dispatchers on terbium for wikidatawiki for a short while
  • 21:24 andrewbogott: restarting nginx on nitrogen
  • 21:22 andrewbogott: restarting puppetdb on nitrogen
  • 20:00 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms (duration: 00m 14s)
  • 19:59 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: deploying refinery to fix banner activity false alarms
  • 19:53 bblack: reboot cp4029-32 (initial reboot post-puppetization)
  • 19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.4
  • 18:36 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 51s)
  • 18:35 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 50s)
  • 18:28 thcipriani@tin: Synchronized wmf-config/abusefilter.php: SWAT: Enable $wgAbuseFilterProfile on ptwiki T177641 (duration: 00m 50s)
  • 18:20 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Echo/modules/styles/mw.echo.ui.PageNotificationsOptionWidget.less: SWAT: mw.echo.ui.PageNotificationsOptionWidget: Fix CSS after changes in OOjs UI T178439 (duration: 00m 51s)
  • 16:50 hashar: contint1001: installed python3 (pending https://gerrit.wikimedia.org/r/385210 )
  • 15:14 marostegui: Stop replication in sync on db1103 and db1072 - https://phabricator.wikimedia.org/T164488
  • 15:02 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 00m 07s)
  • 15:02 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
  • 14:36 mforns@tin: Finished deploy [analytics/refinery@0c9ba04]: (no justification provided) (duration: 03m 29s)
  • 14:33 mforns@tin: Started deploy [analytics/refinery@0c9ba04]: (no justification provided)
  • 13:20 moritzm: upgrading mw2120-mw2147 to wikidiff2 1.5.1
  • 12:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034 and db2036 - T178359 (duration: 00m 50s)
  • 11:40 akosiaris: T177196 upload prometheus-postgres-exporter_0.2.0+ds-2 to apt.wikimedia.org/stretch-wikimedia/main and copied over to apt.wikimedia.org/jessie-wikimedia/main
  • 11:10 moritzm: upgrading mw1276-mw1279 (canary servers) to wikidiff2 1.5.1
  • 10:14 godog: upgrade prometheus-hhvm-exporter to 0.4-1
  • 10:07 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 09:54 ppchelko@tin: Finished deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia (duration: 07m 32s)
  • 09:50 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression (duration: 00m 02s)
  • 09:50 ariel@tin: Started deploy [dumps/dumps@2b9326b]: improved job batches for rev content 7z recompression
  • 09:48 moritzm: installing pyjwt security updates
  • 09:47 ppchelko@tin: Started deploy [restbase/deploy@2001a66]: Enable summary on Cass 3 for all but wikipedia
  • 08:13 marostegui: Stop replication in sync on db1103 and db1072 - T164488
  • 08:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 50s)
  • 07:31 marostegui: Stop MySQL on db2034 and db2036 to clone db2092
  • 06:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 and db2036 - T178359 (duration: 01m 08s)
  • 06:02 marostegui: Compress revision table across db1044 databases (s3)
  • 05:46 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1102 for s6 - T174509 T177772
  • 05:45 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1064 - T174509 T177772
  • 05:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051 - T174509 (duration: 00m 50s)
  • 05:41 marostegui: Optimize pagelinks and templatelinks on db1051 - T174509
  • 05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T174509 (duration: 00m 50s)
  • 05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 - T174509 (duration: 00m 50s)
  • 05:29 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support (take 2), T178441
  • 05:08 awight@tin: Finished deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441 (duration: 00m 18s)
  • 05:08 awight@tin: Started deploy [ores/deploy@19b1851]: Push ORES w/ Celery 4 support, T178441
  • 03:38 bblack: cp3034 - upgraded vhtcpd to 0.1.0 for testing
  • 03:38 bblack: cp3030 - upgraded vhtcpd to 0.1.0 for testing
  • 03:01 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 19 03:01:54 UTC 2017 (duration 7m 2s)
  • 02:54 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 09m 28s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 09m 40s)
  • 00:01 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/385106/2 (duration: 00m 50s)

2017-10-18

  • 23:51 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
  • 23:49 maxsem@tin: Synchronized static/images/project-logos/: https://gerrit.wikimedia.org/r/#/c/385094/2 (duration: 00m 50s)
  • 23:45 maxsem@tin: Synchronized php-1.31.0-wmf.4/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/384903/ (duration: 00m 50s)
  • 23:44 maxsem@tin: Synchronized php-1.31.0-wmf.3/extensions/Collection/: https://gerrit.wikimedia.org/r/#/c/385005/ (duration: 00m 51s)
  • 23:34 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/385007/ (duration: 00m 51s)
  • 23:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.4
  • 23:04 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 22:55 tgr@tin: Synchronized private/: Remove the llama (T150554) (duration: 00m 50s)
  • 22:27 tgr@tin: Synchronized wmf-config/: T174651 / gerrit 384908 - deploy ReadingLists to beta - noop for production (duration: 00m 53s)
  • 20:49 ejegg: updated CiviCRM from 7c26791 to b9d2067
  • 20:16 XioNoX: Changing MTUs on interfaces to NTT
  • 19:19 bblack: uploaded vhtcpd-0.1.0-1 to jessie-wikimedia, testing on cp1008 only for now
  • 19:17 andrewbogott: restarting nodepool
  • 19:15 andrewbogott: stopping nodepool temporarily to give rabbit a chance to catch up
  • 19:01 MaxSem: ran LoginNotify/maintenance/migratePreferences.php on test and test2
  • 18:57 andrewbogott: restarting nova-network on labnet1001
  • 18:42 andrewbogott: restarting rabbitmq-server on labcontrol1001; too many timeouts
  • 16:29 elukey: stop ircecho on einstenium for puppet shower
  • 16:27 herron: restarted apache on puppetmasters
  • 14:23 moritzm: uploaded hhvm 3.18.5+dfsg-1+wmf1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 14:03 chasemp: add static ip to cloud mwstake project per T178012
  • 14:03 chasemp: bump up quota on cyberpower cloud project per T178332
  • 14:03 chasemp: create reading-lists cloud project
  • 13:28 zeljkof: EU SWAT finished!
  • 13:22 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [cirrus] Turn on recall A/B test on enwiki (T177502) (duration: 00m 50s)
  • 13:21 gehel: repooling wdqs1003 now that it has catched up on updates
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: pagePreviews: Restart A/B test on enwiki and dewiki (T176469) (duration: 00m 51s)
  • 13:05 moritzm: uploaded wikidiff2 1.5.1 to apt.wikimedia.org
  • 12:11 mobrovac@tin: Finished deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name (duration: 07m 13s)
  • 12:04 mobrovac@tin: Started deploy [restbase/deploy@3c7abf6]: Bug fix: Remove the space in the summary table name
  • 11:54 mobrovac@tin: Finished deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries (duration: 08m 59s)
  • 11:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to help db1071 because db1082 has crashed (duration: 00m 50s)
  • 11:45 mobrovac@tin: Started deploy [restbase/deploy@99052c1]: Use Cass3 storage for ruwiki summaries
  • 11:31 moritzm: upgrading mw1262-mw1265 (canaries) to wikidiff2 1.5.1
  • 11:26 marostegui: Optimize pagelinks and templatelinks on db1102 s7 - T174509
  • 11:21 moritzm: upgrading mw1261 to wikidiff2 1.5.1
  • 11:11 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2 (duration: 08m 53s)
  • 11:03 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now), part #2
  • 11:02 mobrovac@tin: Finished deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now) (duration: 05m 23s)
  • 10:57 mobrovac@tin: Started deploy [restbase/deploy@15619e0]: Introduce the page/summary proxy (no-op for now)
  • 10:50 moritzm: rebooting multatuli for kernel update
  • 10:21 moritzm: imported linux 4.9.30-2+deb9u5~bpo8+1 for jessie-wikimedia
  • 09:51 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098, depool db1082 (crashed) (duration: 00m 50s)
  • 09:39 moritzm: installing xserver/xvfb security updates
  • 09:36 jynus: reloading dbproxy1010's haproxy configuration
  • 09:30 elukey: drop MobileWikiAppToCInteraction_10375484_15423246 from the log database on dbstore1002,db1047,db1046 - T177960
  • 08:41 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Re-enable Statement usage tracking on cawiki (T151717) (duration: 00m 50s)
  • 08:25 marostegui: Stop MySQL on db2019 to copy its data to db2084 - T178359
  • 08:23 mobrovac@tin: Finished deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis (duration: 01m 17s)
  • 08:22 mobrovac@tin: Started deploy [changeprop/deploy@065a06e]: Bug fix: Apply ignore_errors on a per-request basis
  • 08:09 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable description usage tracking on a few test wikis (T177155) (duration: 00m 50s)
  • 07:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T174509 (duration: 00m 49s)
  • 07:46 marostegui: Optimize pagelinks and templatelinks on db1056 - T174509
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1034 - T174509 (duration: 00m 59s)
  • 07:14 gehel: depooling wdqs1003 until it catches up on updates
  • 07:02 marostegui: Stop MySQL on db2079 to copy its data to db2084 - T178359
  • 06:46 _joe_: powercycling wdqs1003
  • 05:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 - T164488 (duration: 00m 49s)
  • 05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 - T174509 (duration: 00m 50s)
  • 05:40 marostegui: Optimize pagelinks and templatelinks on db1055 - T174509
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 - T174509 (duration: 00m 49s)
  • 05:36 marostegui: Optimize pagelinks and templatelinks on db1098 - T174509
  • 05:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T174509 (duration: 00m 50s)
  • 05:25 marostegui: Optimize pagelinks and templatelinks on db1102 for s2 - T174509
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 18 03:17:01 UTC 2017 (duration 7m 3s)
  • 03:09 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.4) (duration: 14m 58s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 21s)

2017-10-17

  • 23:31 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Citoid/modules/ve.ui.CiteFromIdInspector.css: SWAT: ve.ui.CiteFromIdInspector: Fix CSS for context menus after changes in OOjs UI T178324 (duration: 00m 50s)
  • 23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/widgets/ve.ui.MWMediaInfoFieldWidget.css: SWAT: ve.ui.MWMediaInfoFieldWidget: Fix positioning of icons T178415 (duration: 00m 50s)
  • 23:22 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/shell/Command.php: SWAT: Shell\Command: Better walltime fallback T178314 (duration: 00m 51s)
  • 23:15 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Remove setting no longer in MediaWiki (duration: 00m 50s)
  • 23:06 mutante: disabling Icinga notifications for service recommendation_api on scb hosts - please remember to re-enable once ticket is resolved (T178445) (https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=recommendation_api%20endpoints)
  • 22:28 ejegg: updated CiviCRM from fb513cc to 7c26791
  • 21:55 foks: Removed 2FA from User:Amakuru
  • 21:39 robh: cp4026 being returned to service post memory replacement. puppet runs fine and all icinga checks are green. repooled.
  • 21:23 robh: cp4026 coming down for memory swap, no point in maint moding the system since other cp checks will alert anyhow
  • 20:14 mutante: gerrit2001 - re-enabled puppet, attempting restart with --slave after gerrit:384758
  • 19:54 mutante: cobalt - re-enabled puppet, ran puppet, systemd says gerrit is active(running)
  • 19:51 mutante: cobalt - stopping gerrit service, confirming pid file gets removed, move /etc/init.d/gerrit to /root
  • 19:50 mutante: short gerrit downtime for systemd change coming up
  • 19:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4
  • 19:06 mutante: gerrit2001 - rebooting to verify systemd change and service comes back properly
  • 18:53 mutante: gerrit2001: stopping the gerrit service
  • 18:52 mutante: cobalt: temp disable puppet | gerrit2001: stop puppet
  • 18:22 paravoid: rebooting sodium for kernel/distro upgrade
  • 18:14 paravoid: dist-upgrading sodium to stretch
  • 18:12 demon@tin: Finished scap: bootstrap wmf.4 (duration: 48m 41s)
  • 17:23 demon@tin: Started scap: bootstrap wmf.4
  • 16:49 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 [keeping static files] (duration: 01m 24s)
  • 16:35 gehel: restart tilerator / tileratorui on all maps cluster after deployment of latest fix - T177389
  • 16:34 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389 (duration: 01m 42s)
  • 16:32 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389
  • 15:46 _joe_: killed a stuck gerrit process on cobalt
  • 14:47 bd808: wikitech: Cleaned up 'importers' user_group entries (T171682)
  • 14:25 Amir1: end of cleaning up ores_classification table in plwiki, start of ruwiki (T159753)
  • 14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174509 T177772 (duration: 00m 45s)
  • 14:05 Amir1: start of cleaning up ores_classification table in plwiki (T159753)
  • 14:00 zeljkof: EU SWAT finished
  • 13:59 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Fix typo in UserTesting config var (take 2) (T177502) (duration: 00m 45s)
  • 13:47 zeljkof: EU SWAT continues
  • 13:31 zeljkof: stopping EU SWAT
  • 13:29 jynus: killing commonswiki script on tin
  • 13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: SWAT: Remove the 20 edits threshold from ULSCompactLinksDisablePref.php (duration: 00m 46s)
  • 12:53 volans: upgrading cumin masters to cumin 1.2.2-1
  • 12:40 moritzm: upgrading mwdebug* to hhvm-wikidiff 1.5.1
  • 11:47 marostegui: Optimize recentchanges on db1102 for s4 - T174509
  • 11:47 marostegui: Optimize templatelinks and pagelinks on db1102 for s4 - T174509
  • 11:45 Amir1: end of cleaning up ores_classification table in hewiki, start of nlwiki (T159753)
  • 11:39 Amir1: end of cleaning up ores_classification table in fiwiki, start of hewiki (T159753)
  • 11:32 jynus: test-installing proxysql on wasat T175672
  • 11:24 Amir1: end of cleaning up ores_classification table in fawiki, start of fiwiki (T159753)
  • 11:12 gehel: restarting wdqs-updater on wdqs1004 - T175919
  • 10:40 Amir1: end of cleaning up ores_classification table in etwiki, start of fawiki (T159753)
  • 10:36 Amir1: end of cleaning up ores_classification table in cswiki, start of etwiki (T159753)
  • 10:26 Amir1: start of cleaning up ores_classification in cswiki (T159753)
  • 10:16 akosiaris: removing akosiaris's yubikey from network devices
  • 09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174509 T177772 (duration: 00m 46s)
  • 09:23 marostegui: Stop MySQL on db2084 for multi-instance testing
  • 09:16 mobrovac@tin: Finished deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3 (duration: 07m 34s)
  • 09:08 mobrovac@tin: Started deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3
  • 09:08 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2 (duration: 25m 57s)
  • 08:45 gehel: restarting blazegraph on wdqs1004 for GC tuning (adding -XX:+G1PrintRegionLivenessInfo) - T175919
  • 08:42 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2
  • 08:40 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805 (duration: 01m 05s)
  • 08:39 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805
  • 07:54 marostegui: Stop MySQL and reboot db2081 to see if it works fine - T178140
  • 07:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:42 marostegui: Stop MySQL on db2079 to clone db2086 - T170662
  • 07:36 moritzm: rebooting boron for kernel update
  • 07:31 ema: upgrade misc_eqiad to varnish 5 T177233
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 - T174509 (duration: 00m 45s)
  • 07:11 marostegui: Optimize templatelinks and pagelinks on db1067 - T174509
  • 07:10 marostegui: Optimize enwiki.ores_classification on db1067 - T159753
  • 06:55 moritzm: installing libffi security updates on trusty (Debian already fixed)
  • 06:07 marostegui: Stop replication in sync on db1072 and db1103 for data drift fixing - T164488
  • 05:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T174509 (duration: 00m 45s)
  • 05:54 marostegui: Optimize pagelinks and templatelinks on db1034 - T174509
  • 05:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 - T174509 T177772 (duration: 00m 45s)
  • 05:46 marostegui: Optimize pagelinks, templatelinks and recentchanges on db1085 T177772 T174509
  • 05:42 marostegui: Optimize pagelinks, templatelinks and ores_classification on db1095 - T174509 T159753
  • 05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174509 (duration: 00m 45s)
  • 05:38 marostegui: Optimize recentchanges on db1081 - T177772
  • 05:37 marostegui: Optimize pagelinks and templatelinks on db1081 - T174509
  • 05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 45s)
  • 05:32 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
  • 05:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174509 (duration: 00m 46s)
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 17 02:32:30 UTC 2017 (duration 6m 57s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 23s)
  • 00:34 ejegg: updated CiviCRM from 824c2d8 to fb513cc

2017-10-16

  • 23:49 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/CheckUser/specials/SpecialCheckUser.php: SWAT: Revert "Revert "Restore checkuser-userlinks when exist"" T170507 (duration: 00m 46s)
  • 23:46 ebernhardson: increase elasticsearch log levels for transport to debug to help track down T167410
  • 23:39 ejegg: updated civicrm from b953064 to 824c2d8
  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: ContainmentSet: Use strict comparison for array_search() T177825 (duration: 01m 45s)
  • 20:35 chasemp: disable puppet around cloud things for a refactor merge
  • 20:20 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 (duration: 02m 59s)
  • 19:59 demon@tin: Synchronized wmf-config/CommonSettings.php: Ief0ea01f (duration: 00m 46s)
  • 19:58 jynus: sudo apt install wmf-mariadb101-client on terbium for debugging an issue
  • 19:57 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Ief0ea01f (duration: 00m 46s)
  • 19:56 demon@tin: Synchronized tests/cirrusTest.php: Ief0ea01f (duration: 00m 47s)
  • 18:18 thcipriani@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add configuration for statement indexing for Wikidata T175199 (duration: 00m 47s)
  • 17:35 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files (duration: 00m 03s)
  • 17:35 ariel@tin: Started deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files
  • 17:08 gehel@tin: Finished deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates (duration: 02m 20s)
  • 17:06 gehel@tin: Started deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates
  • 16:56 ottomata: deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223
  • 16:39 joal@tin: Finished deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix (duration: 04m 42s)
  • 16:35 joal@tin: Started deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix
  • 16:23 joal@tin: Finished deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix (duration: 06m 56s)
  • 16:16 joal@tin: Started deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix
  • 16:09 ema: upgrade misc_codfw to varnish 5 T177233
  • 16:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T174509 T177772 (duration: 00m 46s)
  • 15:51 gehel: killing stuck tileshell process on maps-test2001 - T177389
  • 15:51 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389 (duration: 00m 20s)
  • 15:50 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389
  • 15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T174509 (duration: 00m 46s)
  • 15:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174509 T177772 (duration: 00m 47s)
  • 14:34 twentyafterfour: deploying hotfix for T178068 and T178107
  • 14:30 moritzm: uploaded wikidiff 1.5.1 to apt.wikimedia.org for jessie/experimental
  • 14:08 hashar: European swat is complete
  • 14:08 hashar@tin: Synchronized wmf-config/abusefilter.php: (T177760) Enable AbuseFilter profiler | (T177761) Allow AbuseFilter to block & grant relevant permissions to permit administrators to manage the restricted option. (duration: 00m 47s)
  • 13:50 moritzm: upgrading deployment-mediawiki0[56] to wikidiff2 1.5.1
  • 13:45 hashar@tin: Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)
  • 13:44 moritzm: upgrading deployment-mediawiki04 to wikidiff2 1.5.1
  • 13:42 hashar@tin: Synchronized wmf-config: [cirrus] Disable A/B test of MLR on testing on 18 wikis with > 1% of search traffic - T177490 (duration: 00m 48s)
  • 13:38 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add additional namespaces to search results for bnwikisource - T178041 (duration: 00m 49s)
  • 12:59 gehel: restart cassandra on maps1001
  • 12:38 ema: upgrading eqiad LVSs to pybal 1.14.2 (T178149, T177815)
  • 11:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T174509 (duration: 00m 46s)
  • 11:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1070 - T174509 (duration: 00m 46s)
  • 11:10 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 10:48 Amir1: cleaning up ores_classification table in wikidatawiki (T159753)
  • 10:03 ema: upgrading codfw LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:54 ema: upgrading esams LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:44 ema: upgrading ulsfo LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:43 ema: pybal 1.14.2 uploaded to apt.w.o
  • 09:40 moritzm: restarting cassandra on restbase1007 to pick up NSS security update
  • 09:39 moritzm: restarting cassandra on cerium/xenon/praseodymium to pick up NSS security update
  • 09:37 gehel: restarting elasticsearch on elastic1017
  • 09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174509 T177772 (duration: 00m 46s)
  • 09:07 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1088 - https://phabricator.wikimedia.org/T174509 https://phabricator.wikimedia.org/T177772
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174509 T177772 (duration: 00m 46s)
  • 08:53 marostegui: Stop MySQL on db1050 as it will be decommissioned - T178162
  • 08:46 marostegui: Remove db1050 from tendril - T178162
  • 08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)
  • 07:56 marostegui: Stop db1103 and db1038 at the same position to fix data drifts - T164488
  • 07:42 moritzm: installing NSS security updates
  • 07:30 marostegui: Stop db1103 and db1072 at the same position to fix data drifts - T164488
  • 07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 46s)
  • 07:19 marostegui: Optimize table ores_classification on db1073 - T159753
  • 07:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 to the config - T170662 (duration: 00m 46s)
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2084 to the config - T170662 (duration: 00m 47s)
  • 06:25 marostegui: Stop MySQL on db2079 to clone db2084 from it - T170662
  • 06:12 marostegui: Optimize pagelinks and templatelinks on db1094 - T174509
  • 06:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174509 T177772 (duration: 00m 46s)
  • 06:04 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1093 - T174509 T177772
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1070 - T174509 (duration: 00m 46s)
  • 05:59 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1070 - T174509
  • 05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174509 T177772 (duration: 00m 46s)
  • 05:53 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1084 - T174509 T177772
  • 05:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174509 (duration: 00m 46s)
  • 05:47 marostegui: Optimize pagelinks and templatelinks on db1074 - T174509
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174509 (duration: 00m 47s)
  • 05:37 marostegui: Optimize templatelinks and pagelinks on db1073 - T174509
  • 05:36 marostegui: Optimize templatelinks and pagelinks on db1073 - 174509
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 16 02:35:34 UTC 2017 (duration 6m 36s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 07m 44s)

2017-10-15

  • 11:15 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916

2017-10-14

  • 03:52 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)

2017-10-13

  • 23:04 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
  • 20:36 mutante: gerrit2001 - removing hhvm package (which is now possible without also removing scap, thanks thcipriani), apt-get autoremove to remove more unused packages (T178039)
  • 18:52 joal@tin: Finished deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version (duration: 15m 02s)
  • 18:40 joal@tin: (no justification provided)
  • 18:37 joal@tin: Started deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version
  • 16:54 otto@tin: Finished deploy [analytics/refinery@28db253]: (no justification provided) (duration: 05m 50s)
  • 16:48 otto@tin: Started deploy [analytics/refinery@28db253]: (no justification provided)
  • 15:31 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 (duration: 00m 46s)
  • 14:59 mutante: added cparle to wmf LDAP group
  • 14:38 jynus: stopping slave on db1056 and optimize recentchanges
  • 14:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100, depool db1056, fix db1098 weight (duration: 00m 46s)
  • 14:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 (duration: 00m 46s)
  • 13:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 (duration: 00m 47s)
  • 12:57 godog: upload scap 3.7.1-1 - T127762
  • 11:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 11:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 10:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 00m 47s)
  • 10:22 godog: roll-restart pybal to apply https://gerrit.wikimedia.org/r/#/c/383997/ - T178149
  • 10:04 jynus: stopping slave on db1098 and optimize recentchanges
  • 09:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 46s)
  • 09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 09:22 Amir1: a small clean up of ores_classification table in wikidatawiki (T159753)
  • 09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 47s)
  • 09:04 godog: restart pybal on lvs1003 to test logstash depool - T178078
  • 08:57 marostegui: Stop db1103 and db1038 in sync for more checksumming - T164488
  • 08:34 ema: upgrade pybal on lvs1003 to 1.14.1 T177815
  • 08:18 ema: upgrade pybal on lvs1006 to 1.14.1 T177815
  • 08:15 jynus: restarting commons wiki recentchanges purge of wb entries T177772
  • 08:11 ema@neodymium: conftool action : set/pooled=yes; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
  • 08:11 ema@neodymium: conftool action : set/pooled=no; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
  • 08:10 ema: depool/repool logstash1001 (service=logstash-gelf) T178078
  • 07:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2081 to the config - T170662 (duration: 00m 46s)
  • 07:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2081 to the config - T170662 (duration: 00m 47s)
  • 06:23 marostegui: Stop MySQL on db2080 to copy its data to db2081 - T170662
  • 00:22 mutante: authdns servers: deleted /etc/ganglia/conf.d/gndsd.pyconf & /usr/lib/ganglia/python_modules_gdnsd.py - removing ganglia stats

2017-10-12

  • 23:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Restore thanks limit on pl.wikipedia (T169268) (duration: 00m 47s)
  • 23:50 cwd: re-enabled omnimail jobs
  • 23:35 eileen1: update from 2d6668b to b953064
  • 23:25 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Add logging for email blocks (T175419) (duration: 00m 46s)
  • 23:13 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch test wikis to HTML5 fragment mode in links (T152540) (duration: 00m 47s)
  • 22:44 demon@tin: Synchronized php-1.31.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php: silence notices (duration: 00m 47s)
  • 21:36 cwd: disabled omnimail jobs
  • 19:47 ejegg: updated CiviCRM from 47de976 to 2d6668b
  • 19:46 chasemp: disable puppet for lab*services* for merge of 383896
  • 19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.3
  • 18:28 mutante: added Pablo (pgrass) to LDAP group 'wmde' (T177599)
  • 18:15 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/ORES/includes/Hooks.php: Make watchlist queries suck less (duration: 00m 50s)
  • 18:01 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/JsonConfig/includes/JCCache.php: Silence junk warnings (duration: 00m 50s)
  • 15:17 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 15:11 mutante: gerrit2001 - rebooting for kernel upgrade
  • 14:57 mutante: netmon1002 - arming keyholder for rancid deploy
  • 14:52 mutante: netmon1002 (librenms, rancid) - rebooting for kernel upgrade
  • 14:47 moritzm: rebooting kubernetes workers to new 4.9.51 kernel
  • 14:41 moritzm: rebooting kubernetes staging servers to new 4.9.51 kernel
  • 14:29 moritzm: rebooting kubernetes masters to new 4.9.51 kernel
  • 13:54 moritzm: updating image scalers to librsvg 2.40.18
  • 13:48 elukey: deployed the new Analytics Public Druid cluster - T176223
  • 13:45 moritzm: updating remaining thumbor servers to librsvg 2.40.18
  • 13:40 ladsgroup@tin: Synchronized wmf-config/Wikibase.php: SWAT: Remove deprecated config variable for Wikibase (T129475) (duration: 01m 02s)
  • 13:33 moritzm: updating thumbor1001 to librsvg 2.40.18
  • 13:13 moritzm: rolling restart of logstash to pick up thumbor logs (T150734)
  • 13:07 zeljkof: EU SWAT finished
  • 12:56 moritzm: uploaded librsvg 2.40.18 for jessie-wikimedia to apt.wikimedia.org
  • 11:46 ema: upgrading cp3010 to Varnish 5 T177233
  • 11:27 Amir1: finished cleanup of ores_classification in wikidatawiki, still needs more work (T159753)
  • 10:03 ema: lvs1005: upgrade pybal to 1.14.1 T177815
  • 10:02 ema: pybal 1.14.1 uploaded to apt.w.o
  • 09:40 ema: upgrading cp3008 to Varnish 5 T177233
  • 09:06 moritzm: installing nettle update from stretch 9.2 point release
  • 09:01 moritzm: installing ncurses security updates
  • 08:25 moritzm: installing gnutls28 update from stretch 9.2 point release
  • 08:02 Amir1: starting cleanup of ores_classification in wikidatawiki (T159753)
  • 07:51 gehel: reinitialize cassandra on maps-test2004 to test vector tiles - T153282
  • 07:40 moritzm: installing at-spi2-core update from stretch 9.2 point release
  • 07:24 moritzm: installing gnupg2 update from stretch 9.2 point release
  • 07:18 legoktm: purging html5-misnesting linter entries from database on terbium (T178040)
  • 03:25 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 12 03:25:17 UTC 2017 (duration 7m 17s)
  • 03:18 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 15m 57s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 39s)
  • 00:30 twentyafterfour: phabricator update finished (a while ago, I forgot to log it)
  • 00:08 twentyafterfour: upgrading phabricator to #phab-2017-10-11
  • 00:05 mutante: netmon2001 arm keyholder for rancid deploy
  • 00:01 twentyafterfour: Preparing to take phabricator offline for a hopefully momentary upgrade.

2017-10-11

  • 23:55 mutante: netmon2001 - smokeping back up - the reboot, as a side-effect, also resolved the systemd-icinga-alert that was pending in Icinga for a while with comments
  • 23:52 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART II (duration: 00m 50s)
  • 23:50 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART I (duration: 00m 50s)
  • 23:47 mutante: netmon2001 (smokeping) - kernel upgrade, schedule downtime, reboot
  • 23:40 thcipriani@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 49s)
  • 23:38 mutante: netmon1003 - servermon back up - apt-get autoremove to drop unrequired python packages
  • 23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.3/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 50s)
  • 23:35 mutante: netmon1003 - cant seem to reboot normally, attempting via gnt-instance command
  • 23:31 mutante: netmon1003 (servermon) - upgrading kernel (jessie), schedule icinga downtime, rebooting, short downtime for servermon itself
  • 23:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on nowiki T177989 (duration: 00m 50s)
  • 23:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on fawiki T176150 (duration: 00m 50s)
  • 23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/styles/ve.init.MWVESwitchConfirmDialog.css: SWAT: Fix WikiEditor mode switcher widget (duration: 00m 50s)
  • 23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 50s)
  • 23:18 mutante: releases1001 upgrading kernel, installed linux-image-amd64, took out of cache::misc, upgrade libnss3, scheduled downtime, rebooting ...
  • 23:17 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 51s)
  • 22:01 volans: uploaded cumin_1.2.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 21:44 robh: cp4009 shutdown accidental, will boot back up immediately
  • 21:13 mutante: releases2001 - scheduled downtime, rebooting
  • 21:07 mutante: running puppet on cache misc hosts to temp remove codfw backend of releases.wm.org, then rebooting releases2001 for kernel upgrade
  • 20:22 subbu: updated Parsoid to ddf7b293 (T138492, T135667, T177612)
  • 20:18 ssastry@tin: Finished deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293 (duration: 08m 42s)
  • 20:17 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 02m 54s)
  • 20:14 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:13 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 00m 02s)
  • 20:13 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:11 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:10 bsitzmann@tin: Finished deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301) (duration: 05m 25s)
  • 20:09 ssastry@tin: Started deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
  • 20:07 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:05 bsitzmann@tin: Started deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301)
  • 20:04 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:01 chasemp: disable puppet around cloud things to roll refactor out slowly
  • 19:29 mutante: releases1001 - same as 2001, upgraded jenkins to 2.73.2, kept existing config (T177962)
  • 19:27 mutante: releases2001 - upgraded jenkins to 2.73.2, kept existing config (vs overwriting with package config)
  • 19:20 mutante: apt: reprepro copy stretch-wikimedia jessie-wikimedia jenkins
  • 19:16 mutante: releases1001/releases2001 - apt-get autoremove to drop non-required python packages
  • 19:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.3
  • 19:02 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 18:47 gehel: killing stuck osm replication on maps-test2001
  • 18:23 demon@tin: Synchronized wmf-config/Wikibase-labs.php: no-op, labs only (duration: 00m 59s)
  • 18:19 demon@tin: Finished scap: wmf.3 bootstrap (duration: 44m 56s)
  • 17:58 herron: ran /home/faidon/update-netboot-stretch.sh on puppetmaster1001
  • 17:49 moritzm: uploaded jenkins 2.73.2 for jessie-wikimedia to apt.wikimedia.org
  • 17:34 demon@tin: Started scap: wmf.3 bootstrap
  • 17:32 demon@tin: Synchronized php: symlink swap to wmf.2, somehow got missed (duration: 00m 45s)
  • 17:29 demon@tin: Synchronized scap/plugins/prep.py: No-op (duration: 01m 49s)
  • 17:09 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 (duration: 02m 20s)
  • 17:06 demon@tin: Pruned MediaWiki: 1.30.0-wmf.16 (duration: 02m 40s)
  • 16:54 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 [keeping static files] (duration: 01m 44s)
  • 16:24 hasharAway: Upgrade jenkins Maven integration plugin to 3.0 - T177962
  • 16:15 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/380942 - T150734
  • 16:12 hasharAway: Upgrading Jenkins CI T177962
  • 16:09 gehel: decommission maps-test2004 from its cassandra cluster (free a node to test vector tiles) - T153282
  • 16:09 XioNoX: resuming PDU upgrade
  • 15:39 elukey: drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002
  • 15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T174509 (duration: 00m 47s)
  • 15:21 _joe_: stopped redis on ocg* T177931
  • 15:08 _joe_: ocg*: stop ocg; mv /srv/deployment /srv/stale, update-rc.d ocg disable, rm /etc/init/ocg.conf - T177931
  • 14:55 _joe_: manually removing IPVS entries for ocg on eqiad LBs, T177931
  • 14:47 _joe_: rolling restart of low-traffic pybals in eqiad for T177931
  • 14:36 moritzm: installing dbus update from stretch 9.2 point release
  • 13:40 hashar: European SWAT completed
  • 13:39 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove compact language links dblist for simplicity (duration: 00m 46s)
  • 13:38 hashar@tin: Synchronized docroot: Remove compact language links dblist for simplicity (duration: 00m 48s)
  • 13:37 hashar@tin: Synchronized dblists: Remove compact language links dblist for simplicity (duration: 00m 47s)
  • 13:36 hashar@tin: Synchronized wmf-config/CommonSettings.php: Remove compact language links dblist for simplicity (duration: 00m 47s)
  • 13:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Configure wmgBabelMainCategory for the Dinka Wikipedia (duration: 00m 47s)
  • 13:27 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe on eswiki - T177695 (duration: 00m 47s)
  • 13:24 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Enable constraint checks on qualifiers and references - T176863 (duration: 00m 47s)
  • 13:21 moritzm: installing db5.3 security updates
  • 13:03 godog: test graphite-web patch on labmon1001 - T177747
  • 12:54 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717) (duration: 00m 47s)
  • 12:53 marostegui: Start recentchanges purge on s4 primary master T177772
  • 12:48 marostegui: Kill recentchanges purge on s4 primary master - https://phabricator.wikimedia.org/T177772
  • 12:44 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717) (duration: 00m 48s)
  • 12:38 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717) (duration: 00m 47s)
  • 12:32 hoo@tin: Synchronized wmf-config/: Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects (duration: 00m 48s)
  • 12:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgWikibaseDisabledUsageAspects (duration: 00m 47s)
  • 12:27 hoo@tin: scap aborted: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects (duration: 02m 34s)
  • 12:25 hoo@tin: Started scap: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects
  • 11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T174509 (duration: 00m 50s)
  • 11:49 ema: upgrading cp3007 to Varnish 5 T177233
  • 11:34 moritzm: installing ruby1.9 security updates on trusty hosts
  • 11:19 akosiaris: re-enable OTRS stats group T176221
  • 11:11 jynus: starting purge of ruwiki.recentchanges T177772
  • 10:55 akosiaris: OTRS upgrade done T176221
  • 10:44 akosiaris: starting OTRS upgrade T176221
  • 10:24 jynus: stopping dbstore2001:s3 for maintenance T177908
  • 10:19 moritzm: installing curl security updates
  • 09:32 moritzm: installing libxfont security updates
  • 09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T174509 (duration: 00m 46s)
  • 09:27 marostegui: Optimize pagelinks and templatelinks on db1087 - T174509
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T174509 (duration: 00m 47s)
  • 09:04 moritzm: installing curl security updates on app server canaries (along with HHVM restart)
  • 08:50 moritzm: installing ffmpeg security updates
  • 08:38 elukey: reboot kafka-jumbo hosts for kernel updates
  • 08:37 marostegui: Stop replication in sync on db1103 and db1038 to checksum their data - T164488
  • 08:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103 - T164488 (duration: 00m 47s)
  • 07:46 jynus: restart commonswiki recentchanges purging T177772
  • 07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)
  • 07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T174509 (duration: 00m 47s)
  • 07:02 marostegui: Optimize templatelinks and pagelinks on db1086 - T174509
  • 06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T174509 (duration: 00m 47s)
  • 06:57 marostegui: Optimize templatelinks and pagelinks on db1082 - T174509
  • 06:14 marostegui: Set db1068 s4 primary master to read-only OFF - T168661
  • 06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)
  • 06:07 marostegui: Deploy alter table on s4 primary master - T168661
  • 06:01 marostegui: Set read-only on db1068 s4 primary master - T168661
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)
  • 05:48 marostegui: Package update on db1068 s4 primary master - T168661
  • 05:35 marostegui: Disable puppet on db1068 (s4 primary master) - T168661
  • 05:35 marostegui: Disable puppet on db1068 (s4 primary master)
  • 05:13 marostegui: Disable alerts on s4 for 2 hours - T168661
  • 05:06 marostegui: Dump buffer pool on s4 primary master db1068 - T168661
  • 05:03 kartik@tin: Finished deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515 (duration: 02m 57s)
  • 05:00 kartik@tin: Started deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 11 02:32:46 UTC 2017 (duration 6m 39s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 52s)

2017-10-10

  • 23:53 thcipriani@tin: Synchronized wmf-config: SWAT: Disable OCG services PART II T177795 (duration: 00m 48s)
  • 23:52 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable OCG services PART I T177795 (duration: 00m 47s)
  • 23:41 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Collection/RenderingAPI.php: SWAT: Do not request render if renderer not configured T177795 (duration: 00m 47s)
  • 23:39 XioNoX: upgrading ps1-a2-eqiad - T175341
  • 23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: revert Enable Vector print logo on test wiki (duration: 00m 47s)
  • 23:21 thcipriani@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:12 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: wikitech: Align "contentadmin" and "sysop" permissions T171208 (duration: 00m 48s)
  • 23:08 eileen1: ran the group clean up script (reduced smart groups)
  • 22:28 eileen1: changed civicrm smart group cache timeout to 5 civicrm/admin/setting/search?reset=1
  • 21:15 bblack: esams repooling - T167840
  • 19:30 krinkle@tin: Synchronized docroot/noc/: Clean up noc base.css (duration: 00m 48s)
  • 19:07 krinkle@tin: Synchronized docroot/noc/: Clean up noc/index CSS (duration: 00m 47s)
  • 19:00 XioNoX: starting the work for T167840
  • 18:41 mutante: logstash*: delete /usr/lib/ganglia/python_modules/elasticsearch_monitoring.py and /etc/ganglia/conf.d/elasticsearch.pyconf via cumin (T177225)
  • 17:32 XioNoX: depooling esams from DNS for T167840
  • 17:26 gehel: shutting down and restarting elasticsearch on relforge1001 for testing - T170378
  • 17:25 krinkle@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery.ui/: I717f2580e3aae (duration: 00m 47s)
  • 16:19 milimetric@tin: Finished deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder (duration: 16m 01s)
  • 16:03 milimetric@tin: Started deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder
  • 15:48 jynus: starting purge of commonswiki.recentchanges T177772
  • 15:39 thcipriani@tin: Synchronized README: noop deployment to test new logstash-checker.py host (duration: 00m 47s)
  • 15:27 jynus: upgrading neodymium and sarin to mariadb-client 10.1.28
  • 15:23 ema: cp1008 upgraded to varnish 5 T168529
  • 15:00 godog: test stretch 9.2 upgrade on ms-fe2005 - T177739
  • 14:59 cmjohnson1: lvs1007 swapping NIC card
  • 14:53 godog: test stretch 9.2 upgrade on ms-be2013 - T177739
  • 14:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T174509 (duration: 00m 47s)
  • 14:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174509 (duration: 00m 47s)
  • 14:31 marostegui: Optimize table ores_classifications on db1080 - T159753
  • 14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)
  • 14:24 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 00m 47s)
  • 14:22 elukey: add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511
  • 13:56 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 02m 16s)
  • 13:37 aude@tin: Synchronized wmf-config/Wikibase-production.php: Remove test wiki echo configs from Wikibase config (duration: 01m 31s)
  • 13:20 aude@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gawiki (duration: 01m 50s)
  • 13:11 gehel: restarting rsyslog on mw1180 - T177833
  • 11:32 moritzm: downgrading HHVM on deployment-mediawiki04 to HHVM 3.18.2 temporarily (to further narrow down a problem with the new wikidiff package)
  • 11:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T174509 (duration: 05m 02s)
  • 11:31 dcausse: upgrading relforge to elastic 5.5.2
  • 11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1071 - T174509 (duration: 01m 21s)
  • 10:33 akosiaris: T177511 switch druid100[456] to private1-x-eqiad VLANs
  • 10:26 Amir1: start of cleaning up ores_classification in wikidatawiki (T159753)
  • 08:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2080 to the config - T170662 (duration: 00m 46s)
  • 08:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2080 to the config - T170662 (duration: 00m 47s)
  • 07:04 marostegui: Stop MySQL on db2079 to clone db2080 - https://phabricator.wikimedia.org/T170662
  • 06:35 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s3 - T153033
  • 06:27 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s1 - T153033
  • 06:22 marostegui: Stop MySQL on db1038 to transfer its data to db1072 - T172679
  • 06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)
  • 06:05 marostegui: Stop MySQL on db1072 to move it to s3 - T172679
  • 05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T174509 (duration: 00m 47s)
  • 05:53 marostegui: Optimize templatelinks and pagelinks on db1079 - T174509
  • 05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1071 - T174509 (duration: 00m 48s)
  • 05:45 marostegui: Optimize templatelinks and pagelinks on db1071 - T174509
  • 05:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174509 (duration: 00m 47s)
  • 05:38 marostegui: Optimize templatelinks and pagelinks on db1076 - T174509
  • 05:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174509 (duration: 00m 48s)
  • 05:32 marostegui: Optimize templatelinks and pagelinks on db1080 - T174509
  • 02:35 eileen2: update civicrm from 00da4b0 to 47de976
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 10 02:32:57 UTC 2017 (duration 6m 41s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 49s)
  • 01:21 eileen2: update civicrm from 276cbfd to 00da4b0

2017-10-09

  • 22:18 hoo: Removed empty left-over xmldatadumps/public/other/wikibase/wikidatawiki/20171007 (https://dumps.wikimedia.org/wikidatawiki/entities/20171007/)
  • 19:53 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
  • 19:51 hashar: morning swat completed
  • 19:50 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo/maintenance/updatePerUserBlacklist.php: Sync a live hack in Echo updatePerUserBlacklist https://gerrit.wikimedia.org/r/#/c/383182/ - T173475 (duration: 00m 47s)
  • 19:48 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
  • 19:22 hashar@tin: Synchronized wmf-config/CommonSettings.php: REVERT: Enable new print styles on Vector - T169732 (duration: 00m 49s)
  • 19:19 hashar@tin: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 19:17 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: Reapply "Use User Ids instead of User Names for Echo Mute"" - T173475 (duration: 00m 52s)
  • 18:57 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on hi.wikiversity - T177690 (duration: 00m 47s)
  • 18:52 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on dty.wiki - T177688 (duration: 00m 47s)
  • 18:45 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled user group to sd.wikipedia - T177141 (duration: 00m 47s)
  • 18:00 ottomata: stopping druid services on druid1004 (not 1006)
  • 17:59 ottomata: stopping druid services on druid1006
  • 17:28 ottomata: stopping druid services on druid1005
  • 16:34 ottomata: stopping druid services on druid1006
  • 16:34 ottomata: beginning decom and reinstall process for druid1004-1006 -- T176223
  • 15:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174509 (duration: 00m 47s)
  • 13:53 zeljkof: EU SWAT finished
  • 13:53 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/: SWAT: Revert "[tech-debt] Remove usage of FuzzyLikeThis in favor of simple fuzzy match" (T177727) (duration: 00m 57s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix amwikimedia site name (T176042) (duration: 00m 47s)
  • 13:10 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177737) (duration: 00m 47s)
  • 12:47 ema: restart pybal on eqiad load balancers to pick up bgp-med config change T165584
  • 12:38 ema: restart pybal on codfw load balancers to pick up bgp-med config change T165584
  • 12:31 ema: restart pybal on ulsfo load balancers to pick up bgp-med config change T165584
  • 12:26 ema: restart pybal on esams load balancers to pick up bgp-med config change T165584
  • 11:59 joal@tin: Finished deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy (duration: 10m 24s)
  • 11:49 joal@tin: Started deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy
  • 11:42 hashar: Upgrading Jenkins - T168644
  • 11:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 46s)
  • 11:22 moritzm: installing apt updates from stretch 9.2 point release
  • 11:22 marostegui: Optimize ores_classification table on db1083 - T159753
  • 11:19 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Actually disable injecting RC everywhere but commonswiki and ruwiki (duration: 00m 46s)
  • 11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 - T174509 (duration: 00m 47s)
  • 11:08 moritzm: installing whois updates from stretch 9.2 point release
  • 11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174509 (duration: 00m 46s)
  • 10:54 moritzm: installing expect updates from stretch 9.2 point release
  • 10:50 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable injecing RC records for commonswiki and ruwiki (T171027) (duration: 00m 47s)
  • 10:38 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client: Re-instate $wgWBClientSettings[injectRecentChanges] (T171027) (duration: 00m 56s)
  • 10:35 hoo: Started dumpwikidatajson.sh on snapshot1007 after T169680 disruptions
  • 10:30 volans: rebooting snapshot1007 for NFS stuck - T169680
  • 10:21 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s7 - T153033
  • 10:21 volans: rebooting snapshot1006 for NFS stuck - T169680
  • 10:15 elukey: rebooting snapshot1001 for NFS stuck - T169680
  • 10:00 volans: rebooting snapshot1005 for NFS stuck - T169680
  • 09:51 volans: rebooting dataset1001 for NFS stuck - T169680
  • 09:51 ema: test setting bgp med on lvs3001/3003 T165584
  • 09:50 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s5 - T153033
  • 09:43 hashar: restarting Jenkins. Deadlock in SSHSlave plugin that causes memory to leak quite rapidly - T177749
  • 09:34 moritzm: installing vim security updates
  • 09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 - T170662 (duration: 00m 47s)
  • 08:56 volans: disabled puppet on 'stat100[5-6]*,snapshot100[1,5-7]*,dataset1001*' to manually recover from nfsd stuck - T169680
  • 08:52 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s4 - T153033
  • 08:51 marostegui: Optimize pagelinks and templatelinks on db1092 - T174509
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174509 (duration: 00m 47s)
  • 08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 - T174509 (duration: 00m 47s)
  • 08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)
  • 08:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)
  • 07:33 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s2 - T153033
  • 07:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2079 to the config - T170662 (duration: 00m 47s)
  • 07:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2079 to the config - T170662 (duration: 00m 47s)
  • 07:01 ema: cp4022 - backend restart, mailbox lag, cache_upload
  • 06:56 marostegui: Stop MySQL on db2075 to use it to clone db2079 - T170662
  • 06:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 - T170662 (duration: 00m 46s)
  • 06:25 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s6 - T153033
  • 06:14 marostegui: Optimize pagelinks and templatelinks on db1039 - T174509
  • 06:11 marostegui: Optimize pagelinks and templatelinks on db1050 - T174509
  • 06:08 marostegui: Optimize pagelinks and templatelinks on db1096 - T174509
  • 06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174509 (duration: 00m 47s)
  • 05:59 marostegui: Optimize pagelinks and templatelinks on db1091 - T174509
  • 05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174509 (duration: 00m 47s)
  • 05:53 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
  • 05:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 47s)
  • 05:44 marostegui: Optimize pagelinks and templatelinks on db1083 - T174509
  • 05:38 marostegui: Stop MySQL on db1083 to upgrade it
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 47s)
  • 02:48 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 9 02:48:21 UTC 2017 (duration 6m 54s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 13m 49s)

2017-10-08

  • 21:40 hoo: Killed "dumpwikidatardf.sh truthy nt" (wikidata truthy dump) on snapshot1007, got stuck after T169680.
  • 11:28 volans: ack-ed puppet not running on stat100[5-6],snapshot100[1,5-7] due to NFSD stuck on dataset1001 - T169680
  • 08:19 elukey: restart varnish backend on cp4026 to stop 503s

2017-10-07

  • 22:46 reedy@tin: Synchronized php-1.31.0-wmf.2/resources/: T177705 (duration: 00m 48s)
  • 22:44 reedy@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialBlock.php: T177705 (duration: 00m 49s)

2017-10-06

  • 18:58 ejegg: updated CiviCRM from 197b22d to 276cbfd
  • 18:38 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/: Unbreak preference changes (T177524) (duration: 00m 47s)
  • 18:23 arlolra@tin: Finished deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf (duration: 12m 41s)
  • 18:10 arlolra@tin: Started deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
  • 15:30 moritzm: repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
  • 15:25 moritzm: repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
  • 12:54 moritzm: repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
  • 11:46 bblack: cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites
  • 11:34 moritzm: repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
  • 11:21 moritzm: installing gdk-pixbuf security updates
  • 11:10 elukey: rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes
  • 11:03 moritzm: installing git security updates on trusty (Debian already fixed)
  • 10:51 moritzm: upgrading HHVM on script runners to HHVM 3.18.5
  • 10:47 moritzm: upgrade remaining video scalers in codfw to HHVM 3.18.5
  • 10:43 elukey: restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart
  • 10:17 moritzm: upgrade remaining job runners in codfw to HHVM 3.18.5
  • 09:53 elukey: stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart
  • 09:17 hashar: Restarted the CI Jenkins for plugin upgrades
  • 09:01 moritzm: upgrade remaining app servers in codfw to HHVM 3.18.5
  • 08:40 moritzm: upgrade remaining image scalers in codfw to HHVM 3.18.5
  • 08:09 ema: upgrade pybal to 1.14.0 on eqiad primary LVSs
  • 07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s)
  • 07:23 ema: upgrade pybal to 1.14.0 on eqiad secondary LVSs
  • 07:18 moritzm: upgrade remaining API servers in codfw to HHVM 3.18.5
  • 07:11 legoktm: live hacking on mwdebug1002
  • 07:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
  • 06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
  • 06:35 ema: set max_connections=0 on cp3032 varnish-be
  • 06:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)
  • 06:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s)
  • 05:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s)
  • 05:35 marostegui: Stop MySQL on db1076 for an upgrade

2017-10-05

  • 23:40 ejegg: updated CiviCRM from cca1a9d to 197b22d
  • 23:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s)
  • 23:34 XioNoX: upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused)
  • 23:23 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s)
  • 23:21 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s)
  • 22:47 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s)
  • 22:43 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s)
  • 22:39 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s)
  • 22:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s)
  • 22:13 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s)
  • 22:13 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
  • 22:04 bd808@tin: Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) (duration: 00m 33s)
  • 22:04 bd808@tin: Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681)
  • 21:35 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
  • 21:32 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s)
  • 21:22 ejegg: updated CiviCRM from 59e3e00 to cca1a9d
  • 21:13 twentyafterfour: Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges
  • 20:47 twentyafterfour: 1.31.0-wmf.2 is now blocked by T177535
  • 20:28 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358
  • 20:26 twentyafterfour: rolling back to 1.31.0-wmf.1
  • 20:11 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
  • 20:10 twentyafterfour: Promote all wikis to 1.31.0-wmf.2 refs T174358 (currently there are no blockers and no significant logspam)
  • 19:59 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 55s)
  • 19:58 andrewbogott: upgrading wikitech-static to REL1_30
  • 19:55 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 52s)
  • 19:49 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/VisualEditor/extension.json: SWAT: Move 'parentheses' message to MWSaveDialog's modules T177446 (duration: 00m 50s)
  • 19:45 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict: SWAT: Dont register simulate page when not enabled as a BF (duration: 00m 52s)
  • 19:05 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 55s)
  • 18:58 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 56s)
  • 18:53 bblack: cp3* upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 18:38 urandom: restbase-ng: lowering compactionthroughput to 4 (current/5 jbod devices)
  • 18:26 thcipriani@tin: Synchronized wmf-config: SWAT: Enable structured change filters by default on all wikis except FlaggedRevs wikis using non-protection modes T177445 (duration: 00m 54s)
  • 18:13 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on all wikis T124742 (duration: 00m 52s)
  • 17:27 cmjohnson1: disable puppet db1024
  • 16:53 moritzm: upgrade remaining app servers in eqiad to HHVM 3.18.5
  • 16:16 XioNoX: failure test for pfw3-codfw
  • 15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T174054 (duration: 00m 50s)
  • 15:25 marostegui: Pulling out one disk from db1078 - T174054
  • 15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T174054 (duration: 00m 51s)
  • 15:16 marostegui: Pulling out one disk from db1076 - T174054
  • 14:53 hashar: All Jenkins infra is now on Java 8 - T162828
  • 14:52 marostegui: Sanitize db1095 for amwikimedia - T176043
  • 14:48 hashar: Restarting Jenkins
  • 14:27 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
  • 13:57 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717) (duration: 00m 50s)
  • 13:51 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: (no justification provided) (duration: 00m 50s)
  • 13:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Document task reference for amwikimedia logo (T176042) (duration: 00m 54s)
  • 13:43 Amir1: ladsgroup@terbium:~$ mwscript createAndPromote.php --wiki=amwikimedia --createAndPromote=sysop,bureaucrat 'Ladsgroup' "password removed obviously" --force (T176042)
  • 13:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Fix amwikimedia logo (T176042) (duration: 00m 50s)
  • 13:26 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaMessages/i18n/wikimediaprojectnames: (no justification provided) (duration: 00m 49s)
  • 13:15 godog: test upgrade node-exporter in thumbor/swift/services/puppet3-diffs/monitoring projects - T166561
  • 13:11 ladsgroup@tin: Synchronized wmf-config/interwiki.php: Create amwikimedia (T176042) (duration: 00m 51s)
  • 13:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 50s)
  • 12:58 ladsgroup@tin: Synchronized multiversion/MWMultiVersion.php: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:53 ladsgroup@tin: Synchronized static/images/project-logos/: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:52 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:50 ladsgroup@tin: rebuilt wikiversions.php and synchronized wikiversions files: Create amwikimedia (T176042)
  • 12:49 ladsgroup@tin: Synchronized dblists: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:36 ladsgroup@tin: Synchronized dblists: (no justification provided) (duration: 00m 51s)
  • 12:22 moritzm: installing poppler security updates on trusty
  • 11:57 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
  • 11:29 jynus: rebuilding globalimagelinks table on dbstore1001:s4
  • 10:35 hashar: contint2001: switched java to version 8 as well as all the jre/jdk utilities managed by alternatives - T162828
  • 10:26 moritzm: upgrade remaining eqiad API servers to HHVM 3.18.5
  • 10:23 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: REVERT: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 50s)
  • 10:22 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: REVERT: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
  • 10:17 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
  • 10:16 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 51s)
  • 10:14 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/SpecialConflictTestPage/SpecialConflictTestPage.php: Fix SimulateTwoColEditConflict when using a page with ns (duration: 00m 50s)
  • 10:09 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.2 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 50s)
  • 10:07 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.1 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 52s)
  • 10:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174054 (duration: 00m 50s)
  • 09:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 - T174509 (duration: 00m 50s)
  • 08:54 marostegui: Stop MySQL on db1104 to clone db1106 - T172679
  • 08:53 gehel: rolling restart of wdqs to pick up new GC options - T175919
  • 08:41 moritzm: upgrade remaining eqiad job runners to HHVM 3.18.5
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)
  • 08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)
  • 08:29 marostegui: Optimize pagelinks and templatelinks on s5 - db1105 - T174509
  • 08:21 moritzm: upgrade remaining eqiad image scalers to HHVM 3.18.5
  • 08:12 marostegui: Optimize templatelinks and pagelinks on s1 s2 s4 s5 s6 and s7 on labsdb1009 - T174509
  • 08:03 jynus: putting labsdb1009 under maintenance
  • 07:54 moritzm: upgrade mw1221-mw1235 (API servers) to HHVM 3.18.5
  • 07:26 legoktm: running batched query of "DELETE FROM linter WHERE linter_cat=12" on all wikis to clear out mostly bogus html5-misnesting category
  • 07:11 ema: update pybal to 1.14.0 on esams primary LVSs
  • 07:06 ema: update pybal to 1.14.0 on esams secondary LVSs
  • 07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 - T174509 (duration: 00m 52s)
  • 07:01 marostegui: Optimize templatelinks and pagelinks on db1099 - T174509
  • 06:56 moritzm: upgrade mw1238-mw1258 (app servers) to HHVM 3.18.5
  • 06:40 marostegui: Stop MySQL on db1104 to clone db1105 from it - T172679
  • 06:32 marostegui: Optimize templatelinks and pagelinks on db1069 - T174509
  • 06:31 marostegui: Optimize templatelinks and pagelinks on db1053 - T174509
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T174509 (duration: 00m 50s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 49s)
  • 06:24 marostegui: Optimize templatelinks and pagelinks on db1089 - T174509
  • 06:17 marostegui: Optimize s2.templatelinks and enwiki.pagelinks on codfw master db2017 (without replication) - T174509
  • 06:13 marostegui: Optimize enwiki.templatelinks and enwiki.pagelinks on codfw master db2048 (without replication) - T174509
  • 06:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 and db2056 - T174509 (duration: 00m 52s)
  • 05:57 elukey: restart varnish backend on cp3040
  • 05:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 - T174509 (duration: 01m 09s)
  • 05:42 elukey: restart varnish backend on cp3030
  • 02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 5 02:59:18 UTC 2017 (duration 7m 23s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 09m 01s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 09m 51s)
  • 00:06 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: T176652 (duration: 00m 49s)
  • 00:04 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T176652 (duration: 00m 50s)

2017-10-04

  • 23:55 eileen: update process_control to 10161f8
  • 23:53 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/jquery.embedPlayer.js: Fix jqmigrate warning (T169385) (duration: 00m 50s)
  • 23:51 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/ORES/includes/Hooks.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:49 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:36 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilter.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:32 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilterGroup.php: Fix PHP fatal when transcluding Special:Recentchanges (T176236) (duration: 00m 50s)
  • 23:32 Krinkle: Upgrading xhgui on tungsten (operations/software/xhgui.git) from 0.5.2 to 0.8.1 (matching version used in operations/mediawiki-config vendor) - new feature: flamegraphs
  • 23:29 catrope@tin: Synchronized wmf-config/throttle.php: Remove expired rules (duration: 00m 50s)
  • 23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile on ptwiki (T177336) (duration: 00m 51s)
  • 23:19 catrope@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-en.svg: Remove unnecessary id attributes (T175670) (duration: 00m 50s)
  • 23:18 catrope@tin: Synchronized php-1.31.0-wmf.2/skins/MinervaNeue/: Correct feature phone threshold detection (T176286) (duration: 00m 53s)
  • 22:35 ejegg: changed DonationInterface cURL timeout for all processors to 12 sec
  • 21:09 arlolra: Updated Parsoid to 477942c3 (T177115)
  • 21:07 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 21:03 arlolra@tin: Finished deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3 (duration: 10m 01s)
  • 20:53 arlolra@tin: Started deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
  • 20:33 arlolra@tin: Finished deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3 (duration: 00m 58s)
  • 20:32 arlolra@tin: Started deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
  • 20:18 pnorman: Start regenerating map tiles on codfw for z13-z14 - T176252
  • 19:55 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.2 refs T174358 (duration: 00m 48s)
  • 19:55 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.2 refs T174358
  • 19:38 pnorman: Start regenerating map tiles on eqiad for z13-z14 - T176252
  • 19:25 twentyafterfour: Deploying MediaWiki 1.31.0-wmf.2 to Group 1 wikis refs T174358
  • 19:06 bblack: cp2* (all codfw caches): upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 18:39 ottomata: reverting roll out of LVS service druid-analytics.svc.eqiad.wmnet - this wont' work with hosts inside of Analytics VLAN
  • 18:23 niharika29@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 50s)
  • 18:22 niharika29@tin: Synchronized php-1.31.0-wmf.1/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 51s)
  • 18:09 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on most group1 wikis (T124742) (duration: 00m 51s)
  • 17:37 elukey: enabled basic ACLs on the Kafka Jumbo cluster - T173493
  • 17:32 bblack: text@ulsfo - upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 17:32 ottomata: deploying new LVS service for druid-analytics-broker
  • 17:16 moritzm: uploaded icu52 to stretch-wikimedia (co-installable forward port of jessie's ICU for stretch to be used for HHVM)
  • 17:06 bblack: cp4022-26 (now all of upload@ulsfo) - upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 16:58 krinkle@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayer.loader.js: T175506 (duration: 00m 52s)
  • 16:29 bblack: lvs1001-3: base package upgrades, autoremove, +python-requests
  • 16:24 bblack: recdns: upgrade base packages
  • 16:23 volans: deleted 'R:File = /usr/local/bin/wmf-reimage' on matching hosts T176955
  • 16:20 bblack: maelant: upgrade base packages
  • 16:11 bblack: lvs1004-6: base package upgrades, autoremove, +python-requests
  • 16:06 jynus: performing schema change on labsdb1009/10/11
  • 16:01 ejegg: disabled CiviCRM dedupe jobs
  • 15:40 bblack: acamar (recdns@codfw) - base package upgrades
  • 15:39 bblack: lvs1007-12: base package upgrades, autoremove, +python-requests
  • 15:37 bblack: hydrogen (recdns@eqiad) - base package upgrades
  • 15:35 marostegui: Power off db2010 to decommission it - T175685
  • 15:35 bblack: recdns: apt autoremove -> install python-requests
  • 15:32 bblack: caches: apt autoremove -> install python-requests
  • 15:28 marostegui: Disable puppet on db2010 - it will be decommissioned - T175685
  • 15:24 andrewbogott: rebuilding labvirt1016
  • 14:57 ema: cp4025: restart varnish backend
  • 14:47 XioNoX: replacing faulty VC cable on fasw-c-eqiad
  • 14:45 bblack: base package upgrades (+autoremove) on lvs3001-2
  • 14:43 bblack: base package upgrades (+autoremove) on lvs3004
  • 14:41 bblack: base package upgrades (+autoremove) on lvs3003
  • 14:24 bblack: upgrading nginx to 1.13.5-1+wmf1~jessie1 on cp4021 (first live cache, upload@ulsfo)
  • 14:18 bblack: base package upgrades (+autoremove) on lvs2*
  • 14:10 bblack: base package upgrades (+autoremove) on lvs4*
  • 13:47 zeljkof: EU SWAT finished
  • 13:46 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
  • 13:45 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
  • 13:41 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 52s)
  • 13:26 zeljkof: one more commit for EU SWAT
  • 13:26 marostegui: Optimize table ores_classification on enwiki codfw master db2048 with replication - might generate lag - T159753
  • 13:22 marostegui: Optimize enwiki.ores_classification on db2069
  • 13:06 zeljkof: EU SWAT finished
  • 13:05 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177370) (duration: 00m 53s)
  • 12:59 jynus: upgrade and reboot labsdb1009
  • 12:56 marostegui: Optimize templatelinks and pagelinks tables on s2 and s7 - labsdb1010 - T174509
  • 12:38 moritzm: upgrading app servers in codfw to HHVM 3.18.5
  • 12:21 moritzm: upgrading image scalers mw1293-mw1295 to HHVM 3.18.5
  • 12:12 moritzm: upgrading HHVM on deployment servers to 3.18.5
  • 12:10 elukey: added two new mediawiki videoscalers - mw1307/1318
  • 12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet
  • 12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
  • 12:00 gehel: mediawiki now uses the LVS endpoint for logstash - T175242
  • 11:49 moritzm: upgrading job runners mw1161-mw1167 to HHVM 3.18.5
  • 11:34 moritzm: installing ghostscript security updates
  • 11:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s)
  • 10:29 jynus: upgrade and reboot labsdb1011
  • 10:22 jynus: draining labsdb1011 connections
  • 10:18 akosiaris: T177374, fully depool wtp1001-wtp1024
  • 10:18 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
  • 10:03 moritzm: install libidn security updates
  • 10:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s)
  • 09:54 _joe_: restarting hhvm on canaries (both appservers and canaries) T107128
  • 09:47 moritzm: upgrading API app servers mw1189-mw1208 to HHVM 3.18.5
  • 09:45 elukey: rolling restart of aqs nodes to pick up the new logstash lvs config
  • 09:39 hoo: Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest.
  • 09:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)
  • 09:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)
  • 09:25 moritzm: installing ocaml security updates on trusty
  • 09:16 marostegui: Reboot db1087 to pick up new kernel
  • 09:09 _joe_: restarting hhvm on mwdebug*, T107128
  • 09:07 marostegui: Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509
  • 09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s)
  • 08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
  • 08:40 mobrovac@tin: Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s)
  • 08:36 moritzm: upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5
  • 08:30 mobrovac@tin: Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3
  • 08:28 marostegui: Stop MySQL on db2010 as it will be decommissioned - T175685
  • 07:30 jynus: stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB
  • 07:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
  • 07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
  • 06:53 marostegui: Stop MySQL on db1087 to clone db1104 from it - T172679
  • 06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)
  • 06:30 marostegui: Optimize templatelinks and pagelinks tables on db1066 - T174509
  • 06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s)
  • 06:21 elukey: restart varnish backend on cp3043
  • 06:19 marostegui: Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509
  • 06:16 elukey: restart varnish backend on cp3041
  • 06:13 marostegui: Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509
  • 05:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s)
  • 03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s)
  • 03:02 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s)
  • 00:33 ejegg: re-enabled fundraising queue consumers
  • 00:29 ejegg: re-enabled CiviCRM de-dupe jobs
  • 00:24 ejegg: re-enabled civicrm cron and ingenico jobs
  • 00:20 ejegg: re-enabled fundraising stats and export jobs
  • 00:16 ejegg: re-enabled omnimail import jobs
  • 00:10 ejegg: re-enabled fundraising audit jobs
  • 00:07 ejegg: re-enabled pending queue consumer

2017-10-03

  • 23:52 ejegg: stopped all jobs for process-control upgrade
  • 23:46 pnorman: Start regenerating map tiles on eqiad for z0-z12 - T176252
  • 23:20 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175971 T176088 Enable RemexHTML on wikitech and eswikiversity (duration: 00m 51s)
  • 23:11 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Pre-configure cirrussearch for 5.5.x upgrade (duration: 00m 51s)
  • 22:26 bsitzmann@tin: Finished deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519) (duration: 06m 14s)
  • 22:20 bsitzmann@tin: Started deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519)
  • 20:29 pnorman: Start regenerating map tiles on codfw for z0-z12 - T176252
  • 20:17 demon@tin: Synchronized php-1.31.0-wmf.2/extensions/GlobalUserPage/includes/Hooks.php: fix broken thing (duration: 00m 51s)
  • 19:58 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.2
  • 19:33 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 51s)
  • 19:17 demon@tin: Finished scap: bootstrap wmf.2 (duration: 33m 50s)
  • 19:01 twentyafterfour: Starting MediaWiki train for 1.30.1-wmf.2
  • 18:43 demon@tin: Started scap: bootstrap wmf.2
  • 18:25 chasemp: change to private vlan for labweb1001 on asw-b-eqiad
  • 18:24 chasemp: change to private vlan for labweb1002 on asw2-d-eqiad
  • 18:24 chasemp: auth-dnsupdate on bahan for 382027
  • 16:51 ejegg: updated payments-wiki from c2a3d43 to b9859f6
  • 15:57 bblack: upgrading basic packages on role::cache::text
  • 15:50 bblack: upgrading basic packages on role::cache::upload
  • 15:42 bblack: upgrading basic packages on role::cache::misc
  • 15:29 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 06m 27s)
  • 15:24 _joe_: uploading blubber to jessie-wikimedia T175296
  • 15:23 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 15:04 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 05m 14s)
  • 14:59 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 14:56 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 04m 25s)
  • 14:56 bblack: upgrading basic packages on cp1065
  • 14:52 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 13:56 zeljkof: EU SWAT finished
  • 13:55 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/Translate/webservices/CxserverWebService.php: SWAT: Make CxserverWebService forward compatible (duration: 00m 45s)
  • 13:46 godog: upgrade grafana to 4.5.2 on krypton - T175980
  • 13:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust wgNamespacesToBeSearchedDefault for enwikibooks, fawikibooks and hewikisource (T176906 T176908 T176907) (duration: 00m 46s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Limit thanks for new users at pl.wikipedia to 3 per day" (T176174 T169268) (duration: 00m 46s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ShortUrl Extension on hiwikiversity (T177187) (duration: 00m 46s)
  • 13:25 marostegui: Optimize templatelinks and pagelinks tables on s1, s4 and s6 on labsdb1010 - T174509
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage Extension on hiwikiversity (T177188) (duration: 00m 46s)
  • 13:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T174764 (duration: 00m 47s)
  • 10:50 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: (no justification provided) (duration: 08m 38s)
  • 10:41 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: (no justification provided)
  • 08:10 Amir1: cleaning up ores_classification tables (T159753)
  • 08:05 marostegui: Deplo alter table on s3 codfw master (db2018) to add PK to the main tables - T163912
  • 08:02 mobrovac@tin: Finished deploy [restbase/deploy@65f519b]: Create new buckets for feed end points (duration: 09m 57s)
  • 07:52 mobrovac@tin: Started deploy [restbase/deploy@65f519b]: Create new buckets for feed end points
  • 07:20 gehel: killing stuck tileshell and cleaning up /srv/osm_expire on maps-test2001 - T175123
  • 07:05 elukey: restart varnish backend on cp3031 (503s)
  • 07:03 marostegui: Optimize pagelinks and templatelinks tables on labsdb1010 for s5 - T174509
  • 07:01 marostegui: Optimize templatelinks and pagelinks tables on s6 codfw master db2028 (this might generate lag) - T174509
  • 06:46 marostegui: Drop now redundant indexes from pagelinks and templatelinks on s7 - T174509
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)
  • 05:54 marostegui: Optimize templatelinks and pagelinks tables on s5 master codfw (db2023): this will generate lag - T174509
  • 05:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 and db2041 - T174509 (duration: 00m 47s)
  • 03:23 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 [keeping static files] (duration: 01m 30s)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 3 02:30:30 UTC 2017 (duration 6m 39s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 22s)

2017-10-02

  • 23:38 catrope@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/: T177250 (duration: 00m 47s)
  • 23:36 catrope@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FiltersViewModel.js: T177107 (duration: 00m 47s)
  • 23:33 mutante: releases2001 - kill and restart jenkins with systemctl
  • 23:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Update WMF mailing address (T173684) and enable jQuery 3 on Wiktionaries (T124742) (duration: 00m 48s)
  • 23:00 mutante: releases1001 - remove and reinstall jenkins
  • 22:56 Reedy: created newsletter tables on officewiki T176199
  • 19:36 ejegg: turned on CiviMail record creation
  • 18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Change the Turkish Wiktionary logo (T176008) (duration: 00m 46s)
  • 18:46 niharika29@tin: Synchronized static/images/project-logos/: Change the Turkish Wiktionary logo (T176008) (duration: 00m 48s)
  • 17:46 gehel@tin: Finished deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes (duration: 01m 48s)
  • 17:44 gehel@tin: Started deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes
  • 14:08 zeljkof: EU SWAT finished
  • 14:07 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Sandbox menu to Javanese Wikipedia (T176308) (duration: 00m 46s)
  • 14:01 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Newsletter on officewiki (T176199) (duration: 00m 46s)
  • 13:59 zeljkof: extending EU SWAT for 5-10 minutes
  • 13:51 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 45s)
  • 13:50 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
  • 13:44 zfilipin@tin: Synchronized static/images/project-logos/zh_classicalwiki-1.5x.png: static/images/project-logos/zh_classicalwiki-2x.png static/images/project-logos/zh_classicalwiki.png wmf-config/InitialiseSettings.php SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
  • 13:42 godog: roll restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/381045/ - T166699
  • 13:38 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Extension:SandboxLink on the nlwikinews (T177170) (duration: 00m 46s)
  • 13:32 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 45s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 46s)
  • 13:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add eliminator as a priviliged account (T176554) (duration: 00m 46s)
  • 13:17 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 46s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 47s)
  • 12:56 elukey: added two new mediawiki jobrunners - mw131[01]
  • 12:50 marostegui: Optimize tables pagelinks and templatelinks on s5: dbstore2001
  • 11:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 - T174509 (duration: 00m 46s)
  • 10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet
  • 10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet
  • 10:04 mobrovac: restbase restarting on restbase1007 to start logging error bodies for T176728
  • 09:55 jynus: stopping and upgrading labsdb1010
  • 09:27 Amir1: finished the cleanup of ores_classification table in enwiki (T159753)
  • 09:12 godog: roll-restart thumbor to upgrade to 1.6 - T172556
  • 09:06 elukey: forced remount of /mnt/hdfs on notebook1002
  • 08:47 gehel: deploying latest tilerator on maps-test only, fix for T175123
  • 08:46 gehel@tin: Finished deploy [tilerator/deploy@b02bd9a]: (no justification provided) (duration: 00m 22s)
  • 08:46 gehel@tin: Started deploy [tilerator/deploy@b02bd9a]: (no justification provided)
  • 08:46 marostegui: Poweroff db2044 for HW maintenance - T174764
  • 08:40 marostegui: Stop MySQL on db2044 to get it ready to replace its mainboard - T174764
  • 08:29 Amir1: finished the cleanup of ores_classification table in wikidatawiki and starting the enwiki one (T159753)
  • 08:24 jynus: stopping dbstore1001 for maintenance
  • 08:12 marostegui: Optimize tables pagelinks and templatelinks on s5: db1100 (already depooled) - T174509
  • 07:59 marostegui: Drop now redundant indexes from pagelinks and templatelinks from s5 - T174509
  • 07:57 Amir1: starting a round of cleanup in ores_classification table in wikidatawiki
  • 07:41 marostegui: Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311
  • 07:32 marostegui: Optimize table pagelinks and templatelinks on s6: dbstore2001 - T174509
  • 07:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 - T174509 (duration: 00m 46s)
  • 07:20 marostegui: Optimize table pagelinks and templatelinks on s6: db2039 - T174509
  • 07:10 marostegui: Optimize table pagelinks and templatelinks on s1: db2034, db2055 db2062 - T174509
  • 07:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034, db2055 db2062 - T174509 (duration: 00m 46s)
  • 06:55 _joe_: restarting a few API appservers with high cpu load
  • 06:19 elukey: restart varnish backend on cp3040
  • 06:10 elukey: restart varnish backend on cp3033
  • 06:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)
  • 06:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)
  • 02:43 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 2 02:43:33 UTC 2017 (duration 6m 39s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 08m 42s)

2017-10-01

  • 21:30 madhuvishy: powercycling labvirt1015
  • 18:58 hashar: restarted Jenkins. Was stuck somehow
  • 13:50 elukey: restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt.

2017-09-30

  • 20:54 catrope@tin: Synchronized wmf-config/CommonSettings.php: Bump $wgResourceLoaderStorageVersion (T176884) (duration: 00m 47s)
  • 00:37 ejegg: increased PayPal Express Checkout cURL timeout to 12 seconds

2017-09-29

  • 22:12 ejegg: updated CiviCRM from 784756b to 59e3e00
  • 21:19 mutante: labstore2003/2004: closed idle screen sessions
  • 20:27 mutante: mwlog1001, oxygen - closing unused screen session
  • 20:20 mutante: install1002/install2002 - closing unused screen sessions (except a tail -f)
  • 20:11 mutante: restbase2001 - closing screen sessions that weren't running anything anymore
  • 19:23 ebernhardson: completed running cirrussearch category moves on commonswiki
  • 18:19 catrope@tin: Synchronized php-1.31.0-wmf.1/includes/libs/CSSMin.php: T176884 (duration: 00m 47s)
  • 18:08 chasemp: remove 'ToolLabs webservice - lighttpd on precise' catchpoint check for Catchpoint
  • 18:07 chasemp: remove labsdb1002 checks on catchpoint (Toolforge)
  • 18:06 chasemp: remove Labs puppetmaster eqiad catchpoint check (for Toolforge)
  • 18:06 chasemp: remove grid-start-precise catchpoint check
  • 18:04 chasemp: update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc
  • 17:00 ebernhardson: kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index
  • 16:37 andrewbogott: re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel
  • 15:52 andrewbogott: re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel
  • 13:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042 - T174509 (duration: 00m 47s)
  • 13:18 Amir1: starting a round of cleanup in ores_classification table in enwiki (T159753)
  • 13:09 elukey: depool mw1265 (hhvm 3.18.5) - disk filled up
  • 12:43 bblack: removed numa=isolate sys entries manually (runtime + future boots) from cp4021
  • 11:53 bblack: cp4021: reboot for numa mode switch
  • 11:44 elukey: re-enable job runner daemons on mw130[8,9]
  • 11:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s)
  • 11:05 elukey: precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation
  • 09:55 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet
  • 09:33 ema: upgrade pybal to 1.14.0 on codfw primaries
  • 08:20 ema: upgrade pybal to 1.14.0 on codfw secondaries
  • 05:53 ema: cp4024 repooled T174891
  • 05:21 ema: powerup cp4024 T174891
  • 00:26 ejegg: updated fundraising tools from 69a9627 to 6d4b6f3
  • 00:16 ejegg: updated CiviCRM 2138869 to 784756b
  • 00:07 mutante: releases1001 - starting Jenkins setup wizard with generated admin pass
  • 00:05 mutante: releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030

2017-09-28

  • 23:46 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s)
  • 23:45 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
  • 22:18 MaxSem: Deployed fix for T176247
  • 21:30 ejegg: updated CiviCRM from 3f48cb0 to 2138869
  • 21:23 XenoRyet: Updated SmashPig from dc62203 to d056a13
  • 19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
  • 19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798)
  • 19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
  • 19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
  • 19:14 ejegg: updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export
  • 19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
  • 19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
  • 18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
  • 18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
  • 18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
  • 18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
  • 18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
  • 17:58 arlolra: Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832)
  • 17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
  • 17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s)
  • 17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
  • 17:40 moritzm: installing apache updates on cobalt
  • 17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
  • 17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
  • 17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
  • 17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
  • 17:08 bblack: cp4021 - depool->reboot for bios update
  • 17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678]
  • 16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978)
  • 16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
  • 16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
  • 16:01 marostegui: Global rename of CodeCat → Rua - T176985
  • 15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
  • 14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
  • 14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
  • 14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
  • 14:42 marostegui: Remove db1035 from tendril - T176931
  • 14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
  • 14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
  • 13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
  • 13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
  • 13:40 zeljkof: EU SWAT finished
  • 13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
  • 13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
  • 13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
  • 13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
  • 13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
  • 13:12 marostegui: Drop  redundant indexes from pagelinks and templatelinks on s4 - T174509
  • 13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
  • 13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
  • 13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
  • 13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
  • 12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
  • 12:27 ema: reboot cp402[2-6] to disable numa isolation
  • 10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
  • 10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
  • 10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
  • 09:59 elukey: added new mediawiki jobrunner - mw1308
  • 09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
  • 09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
  • 09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
  • 08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
  • 07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
  • 07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
  • 06:59 moritzm: installing apache updates on graphite* hosts
  • 06:28 elukey: restart varnish backend on cp3032
  • 05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
  • 05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
  • 05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
  • 05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
  • 05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
  • 05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
  • 05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
  • 04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
  • 03:57 bblack: ulsfo dns-depooled at ~03:51
  • 03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
  • 03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
  • 02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
  • 00:36 twentyafterfour: upgrading phabricator, expect minimal downtime

2017-09-27

  • 23:39 robh: cp4024 is running hardware testing, leave it alone T174891
  • 23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
  • 22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
  • 19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
  • 19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
  • 19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
  • 18:46 catrope@tin: Finished scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter (duration: 22m 55s)
  • 18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
  • 18:23 catrope@tin: Started scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter
  • 18:21 ejegg: updated payments-wiki from 6104e3d to c2a3d43
  • 17:40 ejegg: updated SmashPig Adyen settings
  • 17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
  • 16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
  • 16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
  • 16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
  • 15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
  • 15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
  • 15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
  • 15:51 ema: lvs4002: upgrade pybal to 1.14.0
  • 15:49 ema: lvs4004: upgrade pybal to 1.14.0
  • 15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
  • 15:24 bblack: rebooting cp402[2356] (bnx2x oops)
  • 15:15 moritzm: revoking salt certs for WMCS
  • 15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
  • 15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
  • 15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
  • 15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
  • 15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
  • 14:58 ema: lvs1007: upgrade pybal to 1.14.0
  • 14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
  • 14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
  • 14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
  • 14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
  • 14:24 mlitn@tin: Started scap: Enable 3D extension
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
  • 14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
  • 14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
  • 14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
  • 13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/. Run puppet as well in a batched execution
  • 13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
  • 13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
  • 13:31 akosiaris: T175242 eventstreams requires manual restart
  • 13:25 akosiaris: T175242 parsoid requires manual restart
  • 13:15 akosiaris: T175242 restbase requires manual restart
  • 13:11 addshore: SWAT done
  • 13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
  • 13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
  • 12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
  • 12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
  • 12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
  • 12:14 moritzm: installing apache updates on einsteinium/tegmen
  • 12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)
  • 12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)
  • 09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
  • 09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
  • 08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
  • 08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
  • 08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
  • 08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
  • 08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
  • 08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
  • 08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:55 marostegui: Optimize pagelinks and template links tables on  db2076 db2067 db2060 - T174509
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
  • 07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
  • 06:57 marostegui: Deploy alter table on s6 codfw - T174509
  • 06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
  • 06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
  • 06:37 ema: restart varnish-be on cp3040
  • 06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
  • 06:33 ema: restart varnish-be on cp3041
  • 06:29 marosteg1i: Reboot db2044 after storage failure - T174764
  • 06:15 elukey: restart varnish backend on cp3043
  • 05:41 madhuvishy: service nova-fullstack restart on labnet1001
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
  • 03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)

2017-09-26

  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
  • 23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
  • 23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
  • 22:03 no_justification: gerrit master restart, back momentarily
  • 20:32 mutante: phab1001 - restarted apache after upgrade
  • 20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
  • 20:20 mutante: phab1001 (prod phab) - upgrade Apache package
  • 20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
  • 19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
  • 18:25 mepps: Updated crm from 89b8007 to 3f48cb0
  • 18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
  • 17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
  • 15:50 volans: powecycled cp2012, no ping or ssh console stuck
  • 15:18 XioNoX: starting eqiad frack switch to new infra - T174218
  • 15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
  • 14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
  • 14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
  • 14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
  • 14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)
  • 14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
  • 13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
  • 13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
  • 13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
  • 13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
  • 13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
  • 13:02 marostegui: Drop redundant indexes from s6 - T174509
  • 13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
  • 12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
  • 12:27 marostegui: Drop redundant indexes from s2 - T174509
  • 12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
  • 12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
  • 12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
  • 11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
  • 11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 11:30 kartik@tin: (no justification provided)
  • 11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
  • 11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
  • 10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
  • 10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
  • 10:20 _joe_: rebooting copper (again)
  • 09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
  • 09:10 _joe_: rebooting copper to verify docker setup is correct
  • 08:46 moritzm: installing apache updates on mw* app servers
  • 08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
  • 08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
  • 08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
  • 08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
  • 08:07 marostegui: Drop table trackbacks from s3 - T175051
  • 07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
  • 07:46 marostegui: Drop trackbacks table from s7 - T175051
  • 07:28 marostegui: Drop trackbacks table from s6 - T175051
  • 07:16 moritzm: installing perl security updates
  • 06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
  • 06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
  • 06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
  • 06:18 marostegui: Drop table trackbacks from s4 - T175051
  • 06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
  • 05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
  • 05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)

2017-09-25

  • 23:58 ejegg: updated payments-wiki from d2ce697 to 6104e3d
  • 23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562)
  • 22:53 mutante: gerrit2001 - PXE booting
  • 22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
  • 20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030)
  • 20:37 arlolra: Updated Parsoid to 4230c27b (T176425, T173029)
  • 20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b (duration: 16m 52s)
  • 20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
  • 19:58 XioNoX: renumbering ams BGP communities - T167840
  • 19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
  • 19:02 robh: cp4021 coming down for memory swap
  • 19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
  • 18:54 MaxSem: started populating ip_changes on group2 wikis
  • 18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
  • 18:22 ejegg: updated CiviCRM from 45c56cf to 89b8007
  • 17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
  • 17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
  • 17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
  • 17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
  • 16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
  • 15:57 urandom: T169940: Converting restbase-ng data tables to size-tiered compaction
  • 14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
  • 14:31 gehel: rolling restart of logstash for config change - T176430
  • 14:13 gehel: rolling restart of logstash for config change - T176430
  • 13:46 zeljkof: EU SWAT finished
  • 13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
  • 13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
  • 13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
  • 13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
  • 12:52 moritzm: removed salt master from neodymium
  • 12:38 moritzm: revoked all salt keys on neodymium
  • 11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
  • 11:54 moritzm: removing salt-minion/salt-common from production
  • 11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013)
  • 11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013)
  • 11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572) (duration: 00m 46s)
  • 11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
  • 11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
  • 11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013) (duration: 00m 46s)
  • 11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013) (duration: 00m 45s)
  • 11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013)
  • 11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013) (duration: 00m 46s)
  • 10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
  • 10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
  • 09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
  • 09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
  • 09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
  • 08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
  • 08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
  • 08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
  • 08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
  • 07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
  • 07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
  • 07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
  • 06:38 moritzm: installing apache security updates on logstash*
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)

2017-09-24

  • 14:23 jynus: restarting db2047 for kernel upgrade
  • 14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
  • off: restarted ircecho on einsteinium

2017-09-23

2017-09-22

  • 23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
  • 22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
  • 22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
  • 22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
  • 18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
  • 15:21 hashar: Restarted Jenkins. Out of memory)
  • 14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
  • 14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
  • 14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
  • 13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
  • 12:02 hashar: apt-get upgrade on contint1001 / contint2001
  • 09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
  • 09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
  • 06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)

2017-09-21

  • 23:56 ejegg: updated payments-wiki from 7ed7243 to d2ce697
  • 23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047: Stop delivering search relevance survey javascript (duration: 00m 46s)
  • 23:18 ebernhardson@tin: Synchronized wmf-config: T175047: Turn off human search relevance survey (duration: 00m 48s)
  • 23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047: Turn off human search relevance survey (duration: 00m 46s)
  • 22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
  • 22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 22:10 ejegg: updated payments-wiki from 9da8482 to 7ed7243
  • 21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
  • 21:04 ejegg: updated CiviCRM from fc56901 to 45c56cf
  • 19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
  • 19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
  • 19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
  • 19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
  • 19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
  • 18:56 urandom: T171772: Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
  • 18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
  • 18:41 urandom: T171772: Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
  • 18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
  • 18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
  • 14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
  • 13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
  • 13:12 zeljkof: EU SWAT finished
  • 13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
  • 13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
  • 12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
  • 12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
  • 12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
  • 11:59 moritzm: upgrading apache on deployment servers and script runners
  • 11:40 moritzm: upgrading apache on canary app servers
  • 11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
  • 11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
  • 11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
  • 10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
  • 09:59 dcausse: reindexing group1 wikis T176397
  • 09:54 dcausse: reindexing group0 wikis T176397
  • 08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254)
  • 08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
  • 08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
  • 08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
  • 08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
  • 08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
  • 08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
  • 08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
  • 07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
  • 07:31 _joe_: restarted aphlict on phab1001
  • 07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
  • 07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
  • 07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
  • 06:59 ema: bounce pybal on lvs1006 to clear stale alert
  • 06:55 ema: bounce pybal on lvs1009 to clear stale alert
  • 03:24 bblack: depooled cp4026
  • 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
  • 03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
  • 00:41 eileen: update civicrm from da833ac to fc56901
  • 00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
  • 00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
  • 00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s)
  • 00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s)

2017-09-20

  • 23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
  • 23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
  • 23:40 mutante: phabricator: aphlict (notification service) now running on prod server
  • 23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
  • 23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s)
  • 23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s)
  • 23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
  • 23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s)
  • 23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
  • 22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
  • 21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
  • 21:45 eileen: update civicrm from 0fb4676 to da833ac
  • 21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
  • 20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
  • 20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s)
  • 20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
  • lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
  • 20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263)
  • 20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
  • 19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
  • 19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
  • 19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
  • 19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
  • 19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
  • 19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
  • 19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
  • 19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
  • 19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
  • 19:02 catrope@tin: Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s)
  • 18:40 catrope@tin: Started scap: Patches for T176302, T175962, T173533
  • 17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
  • 17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
  • 17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
  • 17:06 ejegg: removed limit from donation queue consumer
  • 17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
  • 17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
  • 16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
  • 16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
  • 16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
  • 16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
  • 16:34 gehel: cleanup jar hell and restart elastic1051
  • 16:19 gehel: starting elasticsearch on all nodes on eqiad
  • 16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
  • 16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
  • 16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
  • 15:46 ejegg: limited donations queue consumer to 100 per run
  • 15:36 ejegg: updated payments-wiki from 896f16a to 9da8482
  • 15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
  • 15:33 ejegg: limited donations queue consumer to 200 per run
  • 14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
  • 13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
  • 13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
  • 13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 3/3) (duration: 00m 48s)
  • 13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 2/3) (duration: 00m 49s)
  • 13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 1/3, no-op in prod) (duration: 00m 49s)
  • 12:51 ema: restarted varnish-be on cp4028
  • 12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
  • 12:19 ema: restarted varnish-be on cp4027
  • 12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
  • 12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)
  • 11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
  • 11:56 ema: restarted varnish-be on cp4018
  • 11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
  • 11:49 jynus: stopping replication on db1101 for faster repartition work T176311
  • 11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
  • 11:34 ema: restarted varnish-be on cp4010
  • 11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
  • 11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
  • 11:03 moritzm: rolling out new debdeploy release across the fleet
  • 11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
  • 11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:02 ema: restarted varnish-be on cp4008
  • 11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
  • 11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
  • 11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
  • 11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 09:55 bblack: restarted varnish-be on cp3042
  • 09:29 bblack: restarted varnish-be on cp1052
  • 09:23 ema: restarted varnish-be on cp3043
  • 09:21 ema: restarted varnish-be on cp3041
  • 09:19 ema: restarted varnish-be on cp3032
  • 09:14 ema: restarted varnish-be on cp3040
  • 08:42 bblack: restarted varnish-be on cp1053
  • 08:41 bblack: restarted varnish-be on cp1068
  • 08:40 bblack: restarted varnish-be on cp1054
  • 08:40 bblack: restarted varnish-be on cp1065
  • 07:11 ema: cp3030 varnish-be restart due to 503s
  • 03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
  • 00:55 bblack: cp1066 - backend restart, mailbox lag, text
  • 00:54 bblack: cp4016 - backend restart, mailbox lag, text

2017-09-19

  • 23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
  • 23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
  • 23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
  • 23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
  • 23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
  • 23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
  • 23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
  • 23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
  • 23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
  • 22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
  • 22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
  • 22:21 eileen: update civicrm from 9677cf7 to 0fb4676
  • 22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
  • 22:02 mutante: gerrit2001 - systemctl start gerrit
  • 21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
  • 21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
  • 21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
  • 21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
  • 21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
  • 21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
  • 21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
  • 21:06 ema: cp4009: varnish-be restart, 503 spike
  • 20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
  • 20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
  • 19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
  • 19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
  • 19:17 ejegg: updated payments-wiki from 26b16ea to 896f16a
  • 19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
  • 18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
  • 18:35 urandom: T160570, T169940: Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
  • 18:26 arlolra: Updated Parsoid to 05a0965 (T122965, T175101, T173384, T173061, T172896, T174977, T159894)
  • 18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 48s)
  • 18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 49s)
  • 18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 50s)
  • 18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 50s)
  • 18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 12m 26s)
  • 17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
  • 17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
  • 17:43 jynus: stopping db2010 and cloning to db2078
  • 17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
  • 16:45 demon@tin: Started scap: bootstrap wmf.19
  • 16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
  • 16:42 demon@tin: Started scap: bootstrap wmf.19
  • 16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
  • 16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
  • 16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
  • 16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
  • 16:29 jynus: disable puppet on db1009
  • 16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
  • 16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642) (duration: 00m 59s)
  • 16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
  • 16:11 ema: pybal 1.14.0 uploaded to apt.w.o
  • 15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765 (duration: 20m 32s)
  • 15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765
  • 15:22 jynus: enabling firewall on db1020 (m2-master)
  • 15:01 ema: restart varnish-be on cp1055, 503 spike
  • 14:55 jynus: deploying firewall to m5 hosts
  • 14:39 jynus: disabling puppet on all misc active mysql hosts
  • 13:35 zeljkof: EU SWAT finished
  • 13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
  • 13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
  • 13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
  • 13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
  • 12:55 moritzm: rebooting kubestage1001
  • 11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
  • 11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 02m 42s)
  • 11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 10:53 volans: restarted nagios-nrpe-server on stat1005
  • 10:14 jynus: shutting down mysql on db1018
  • 09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
  • 09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
  • 09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
  • 09:30 ema: cp1062 - varnish-backend-restart, mbox lag
  • 08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
  • 08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
  • 08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
  • 08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
  • 07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 00m 37s)
  • 07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 07:50 moritzm: installing bind9 regression update on trusty servers
  • 07:35 kartik@tin: (no justification provided)
  • 07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 01m 04s)
  • 07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
  • 01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
  • 01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
  • 01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
  • 01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
  • 00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
  • 00:10 eileen: update civicrm from 187238f to b6f7dc3

2017-09-18

  • 23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
  • 23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
  • 22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
  • 21:57 bawolff: deployed patch T175900
  • 21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
  • 21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
  • 21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
  • 20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
  • 20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
  • 20:17 arlolra@tin: (no justification provided)
  • 20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 01m 53s)
  • 20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
  • 19:29 XioNoX: restarting pfw3-codfw for software upgrade
  • 19:10 urandom: T160570: Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
  • 19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
  • 19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
  • 19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
  • 19:02 no_justification: gerrit: restarting service, back in a moment
  • 19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
  • 18:56 urandom: T160570: Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
  • 18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
  • 18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
  • 18:37 smalyshev@tin: (no justification provided)
  • 18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
  • 18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
  • 18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
  • 17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
  • 17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
  • 17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
  • 16:46 jynus: shuting down db1100 T175973
  • 16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
  • 16:05 bblack: cp1063 - backend restart, mailbox lag
  • 15:38 bblack: cp1072 - varnish backend restart, mailbox lag
  • 15:28 bblack: cp1074 - backend restart for mailbox lag
  • 15:25 bblack: cp1050 - backend restart for mailbox lahg
  • 14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
  • 13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
  • 13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
  • 13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
  • 13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
  • 13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
  • 13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
  • 12:59 moritzm: installing libxen updates
  • 12:36 moritzm: installing tomcat8 security updates
  • 10:40 ema: cp1099 - backend restart, mailbox lag
  • 06:42 moritzm: installing freexl security updates
  • 06:18 TimStarling: testing EtcdConfig in beta cluster
  • 05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)

2017-09-17

  • 12:49 _joe_: taking a full debug dump of mw1288 after depooling it
  • 12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
  • 09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
  • 08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag

2017-09-16

  • 22:05 godog: compress older otrs directories to reclaim inodes - T171490
  • 16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
  • 15:40 andrewbogott: rebooting labvirt1017
  • 15:34 andrewbogott: rebooting labvirt1015 for T176044

2017-09-15

  • 21:50 mutante: contint2001 - systemctl start zuul
  • 21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
  • 21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
  • 21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
  • 15:46 bblack: cp1049 - backend restart, mailbox lag
  • 15:40 bblack: cp1099 - backend restart, mailbox lag
  • 15:38 bblack: cp1072 - backend restart, mailbox lag
  • 15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
  • 13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
  • 13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
  • 13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
  • 13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
  • 11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
  • 09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
  • 08:54 moritzm: installing libbluetooth updates
  • 08:03 jynus: dropping mariadb analytics users on db1031, db1029
  • 07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601)
  • 07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
  • 07:35 gehel: depooling elastic1020 - T175951
  • 07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
  • 00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)

2017-09-14

  • 23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
  • 23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868) (duration: 00m 47s)
  • 23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
  • 23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
  • 23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
  • 23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
  • 23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095) (duration: 00m 46s)
  • 23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
  • 23:18 mutante: acamar is back up and running - no failed units
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903) (duration: 00m 46s)
  • 23:12 mutante: acamar - staged BIOS upgrade in progress
  • 23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
  • 22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
  • 22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
  • 22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
  • 22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
  • 21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
  • 20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339 (duration: 00m 45s)
  • 20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88 (duration: 00m 46s)
  • 20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df (duration: 00m 46s)
  • 20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
  • 20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
  • 19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
  • 19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
  • 19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
  • 19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
  • 18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
  • 18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
  • 18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
  • 18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
  • 18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
  • 17:57 bblack: cp2001 backend restart, mailbox lag
  • 17:56 bblack: cp1074 backend restart, mailbox lag
  • 17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
  • 17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
  • 17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
  • 16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
  • 15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
  • 15:51 bblack: cp1064 backend restart, mailbox lag
  • 15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
  • 14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898) (duration: 00m 49s)
  • 13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
  • 13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
  • 13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
  • 11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
  • 11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
  • 09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
  • 09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
  • 09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
  • 09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
  • 09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
  • 09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
  • 09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
  • 08:13 gehel: merging cassandra refactoring for puppet - T171704
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
  • 02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
  • 00:46 MaxSem: restarted populateIpChanges to use updated code
  • 00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
  • 00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
  • 00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
  • 00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
  • 00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
  • 00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
  • 00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13

2017-09-13

  • 23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918) (duration: 00m 49s)
  • 23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316) (duration: 00m 57s)
  • 23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
  • 23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413) (duration: 00m 50s)
  • 22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
  • 22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
  • 21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
  • 21:23 ejegg: updated payments-wiki from ed2e481 to 26b16ea
  • grrrr: populating ip_changes on group1 wikis
  • 20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
  • 20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
  • 19:39 andrewbogott: rebooting labtestvirt2001
  • 19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
  • 19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
  • 19:21 urandom: T172384: Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
  • 19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
  • 19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
  • 19:11 urandom: T172384: Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
  • 18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
  • 18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
  • 18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
  • 18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
  • 18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
  • 17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
  • 16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
  • 16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
  • 16:28 gehel: starting decommissioning of wdqs100[12] - T175595
  • 16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
  • 16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
  • 16:22 andrewbogott: rebooting labtestvirt2002
  • 16:08 ottomata: restarting druid-brokers with increase in query cache size
  • 15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
  • 15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
  • 15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
  • 15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
  • 15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038 (duration: 00m 52s)
  • 14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
  • 14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
  • 14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
  • 14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
  • 13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:30 urandom: T169936: Converting RESTBase dev environment to size-tiered compaction
  • 13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
  • 13:23 gehel: initial installation of logstash100[7-9] - T175045
  • 13:21 zeljkof: EU SWAT finished
  • 13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
  • 12:14 moritzm: rebooting mc1015 (spare) for some tests
  • 12:07 jynus: stopping wikidata rebuild maintenance script on terbium
  • 12:07 bblack: cp1049 - backend restart, mailbox lag
  • 11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
  • 11:39 jynus: disabling semi-sync replication on the largest s5 database servers
  • 11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
  • 11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
  • 11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
  • 10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
  • 10:52 volans: reboot lvs3001
  • 10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
  • 09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
  • 09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
  • 08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
  • 08:24 moritzm: installing emacs security updates
  • 07:56 akosiaris: bounce varnish on cp1062, mailbox lag
  • 07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705)
  • 07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
  • 05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36 (duration: 02m 56s)
  • 05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
  • 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
  • 03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
  • 03:03 jynus: dropping old users from phabricator database
  • 03:00 eileen: update civicrm from ee7dda3 to 187238f
  • 02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
  • 01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
  • 00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)

2017-09-12

  • 23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
  • 23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
  • 22:17 chasemp: labvirt1018:~# /sbin/reboot
  • 22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305) (duration: 04m 28s)
  • 21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305)
  • 21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
  • 21:37 MaxSem: Populating ip_changes in group0 wikis
  • 21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
  • 21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
  • 20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
  • 20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
  • 19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
  • 19:45 demon@tin: Started scap: bootstrap wmf.18
  • 19:05 papaul: maintenance complete on mw2256
  • 17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
  • 17:40 cwd: turned on dedupe
  • 16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
  • 16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
  • 16:40 ejegg: updated CiviCRM from 63288ae to ee7dda3
  • 16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
  • 16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
  • 16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
  • 16:18 ejegg: updated SmashPig from dce4f0a to dc62203910
  • 15:43 _joe_: restarted ircecho on einsteinium
  • 15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
  • 15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
  • 15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
  • 15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
  • 14:53 papaul: shutting down mw2256 for maintenance
  • 14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
  • 14:42 ema: cp1074 - backend restart, mailbox lag
  • 14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
  • 14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
  • 14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
  • 14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
  • 14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
  • 14:06 volans: testing reimage of mw2100 - T166300
  • 14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
  • 14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
  • 14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
  • 13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
  • 13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
  • 13:37 gehel: pooling wdqs100[45] - T171210
  • 13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
  • 13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
  • 13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
  • 13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
  • 13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
  • 13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
  • 13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
  • 09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
  • 09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
  • 09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
  • 08:34 gehel: deploying elasticsearch plugins on relforge - T175159
  • 08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
  • 07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
  • 02:10 bblack: cp1050 - backend restart, mailbox lag
  • 02:08 bblack: cp1099 - backend restart, mailbox lagh
  • 00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
  • 00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart

2017-09-11

  • 23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
  • 22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
  • 22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281, no-op in prod) (duration: 00m 45s)
  • 21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
  • 21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
  • 21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
  • 21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
  • 20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
  • 19:11 chasemp: adjust nodepool rate to 10
  • 19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
  • 19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
  • 18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
  • 18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
  • 18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
  • 18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
  • 18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
  • 18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
  • 18:10 robh: scs-ulsfo has successfully updated firmware T174475
  • 18:03 robh: upgrading firmware on scs-ulsfo
  • 17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
  • 17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
  • 17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
  • 15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
  • 15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
  • 14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
  • 14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
  • 14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
  • 14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
  • 13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
  • 13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
  • 13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
  • 13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
  • 13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
  • 11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
  • 10:30 jynus: restarting mysql @ labsdb1001
  • 09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
  • 09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
  • 09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
  • 08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992 (duration: 00m 07s)
  • 08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992
  • 08:16 volans: reimage script tests continue on mc100[1-2] T166300
  • 07:59 moritzm: installing systemd updates from stretch point update
  • 07:41 moritzm: installing remaining libonig security updates
  • 02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)

2017-09-10

  • 21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
  • 13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
  • 13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
  • 00:37 chasemp: labnodepool1001:~# service nodepool restart
  • 00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues

2017-09-09

  • 23:42 chasemp: labnodepool1001:~# service nodepool start
  • 23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
  • 22:59 chasemp: restart nova-api and nova-network
  • 22:49 chasemp: labnodepool1001:~# service nodepool stop
  • 22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
  • 20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
  • 01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740: Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)

2017-09-08

  • 23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740: Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
  • 23:38 mutante: netmon1002 - terminated unused screen session
  • 23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
  • 21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
  • 20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
  • 20:14 mutante: heze - rebooting for firmware upgrade
  • 20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850)
  • 20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
  • 19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650)
  • 19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650)
  • 18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
  • 16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338, followup (duration: 00m 46s)
  • 16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
  • 15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
  • 15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
  • 15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
  • 15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
  • 15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
  • 14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
  • 12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
  • 12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
  • 10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
  • 10:12 _joe_: stopped all md devices syncs on conf200* machines
  • 10:09 _joe_: etcd cluster lost consensus T162013
  • 10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
  • 09:58 _joe_: running the same command on conf2002 too
  • 09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
  • 09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
  • 09:09 moritzm: installing libonig security updates
  • 09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
  • 09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
  • 09:01 moritzm: installing remaining gnutls updates from jessie point release
  • 08:13 moritzm: installing libgd2 security updates
  • 08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
  • 08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
  • 07:53 moritzm: installing ruby2.3 security updates
  • 07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
  • 07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
  • 07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
  • 07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
  • 00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)

2017-09-07

  • 23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
  • 23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
  • 23:15 XenoRyet: Updated Smashpig from 8eb98c1 to dce4f0a
  • 22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
  • 22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
  • 21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
  • 21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
  • 21:01 urandom: T169940: Disabling puppet and restbase service in RESTBase dev
  • 21:00 urandom: T169940: Disabling changeprop in RESTBase dev environment
  • 19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
  • 18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
  • 18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
  • 18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
  • 17:55 ejegg: disabled queue consumers for data cleaning job
  • 17:52 ejegg: updated CiviCRM from c1ece1e to 63288ae
  • 17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
  • 17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
  • 17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
  • 17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
  • 17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
  • 17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
  • 16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
  • 15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
  • 15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
  • 14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
  • 14:40 jynus: rebooting and upgrading es1019
  • 13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
  • 13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
  • 13:14 zeljkof: EU SWAT finished
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
  • 13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
  • 11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
  • 09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
  • 08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
  • 08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
  • 08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
  • 07:57 elukey: force re-mount of /mnt/hdfs on stat1005
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
  • 07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
  • 07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
  • 07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
  • 07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
  • 07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
  • 07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
  • 06:07 eileen: civicrm updated from 73a24b4 to c1ece1e
  • 03:13 eileen: seconds behind=78
  • 03:08 eileen: seconds behind 55
  • 02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
  • 02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
  • 02:43 eileen: restarting groupmember.load after hacking out cache delete calls
  • 02:40 eileen: update civicrm from 764bfe1 to 73a24b4
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
  • 02:05 eileen: replag 217 behind
  • 01:56 eileen: replag 197 seconds
  • 01:47 eileen: replag 23 seconds
  • 01:29 eileen: replag 0
  • 01:22 eileen: replag 550 sec behind master
  • 01:13 eileen: replag delay 422
  • 01:12 eileen: job killed
  • 00:49 eileen: 8389 contacts now
  • 00:49 eileen: 7971 contacts so far
  • 00:46 eileen: started Omnigroupmember.load
  • 00:24 twentyafterfour: phabricator update deployed
  • 00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
  • 00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
  • 00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/

2017-09-06

  • 23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to all webservers refs T174387 (duration: 00m 49s)
  • 23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to wmf.17 refs T174387 (duration: 00m 49s)
  • 23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
  • 23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
  • 23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
  • 23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
  • 23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 Bug: T174106 (duration: 00m 52s)
  • 22:18 no_justification: prior deploy was no-op
  • 22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
  • 22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
  • 22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)
  • 22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)
  • 21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
  • 21:28 ejegg: updated payments-wiki from bf97cd2 to ed2e481
  • 21:24 ejegg: updated CiviCRM from f9c373e to 764bfe1
  • 21:17 ottomata: rebooting kafka-jumbo1004
  • 20:34 arlolra: Updated Parsoid to f9d367ea (T169342)
  • 20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea (duration: 08m 27s)
  • 20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
  • 19:53 ottomata: reimaging kafka-jumbo* with stretch
  • 19:48 chasemp: reboot labvirt1018
  • 18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
  • 18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
  • 18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
  • 18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
  • 18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
  • 18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
  • 18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
  • 18:09 ejegg: rolled payments-wiki back to bf97cd2
  • 18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
  • 18:05 ejegg: updated payments-wiki from bf97cd2 to 6911158
  • 17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
  • 17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
  • 17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156)
  • 17:16 mutante: modify-ldap-group on terbium is broken
  • 17:02 ejegg: disabled creating CiviMail records for Thank You emails
  • 16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa (duration: 01m 01s)
  • 16:41 chasemp: disable puppet for cloud things to test changes
  • 16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
  • 16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
  • 15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
  • 15:53 akosiaris: powercycling lvs3001
  • 15:28 papaul: firmware upgrade on mw2256
  • 15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
  • 15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
  • 15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
  • 14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
  • 13:19 jynus: restarting and upgrading db2040
  • 13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
  • 13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
  • 13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
  • 12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
  • 11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
  • 11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
  • 11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
  • 11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
  • 11:20 godog: reimage restbase1008 with cassandra 3 - T169939
  • 11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
  • 10:26 moritzm: installing perl update from stretch 9.1 point release
  • 10:25 moritzm: installing perl update from jessie 8.9 point release
  • 10:12 godog: reimage restbase1010 with cassandra 3 - T169939
  • 09:39 moritzm: installing libgd security updates
  • 09:28 moritzm: installing libonig security uodates#
  • 09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
  • 08:53 godog: reimage restbase1009 with cassandra 3 - T169939
  • 08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
  • 08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
  • 07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
  • 06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705)
  • 06:37 marostegui: Truncate l10n_cache table across production - T150306
  • 04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
  • 03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
  • 03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
  • away: disabled IPN messages at the PayPal console
  • 00:14 Reedy: that was T174747 Adjust language icon color
  • 00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
  • 00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
  • 00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)

2017-09-05

  • 23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
  • 23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
  • 23:33 foks: 2FA disabled for Omshivaprakash (T175075)
  • 22:17 robh: rebooting mw1163 as its serial console is locked up
  • 22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
  • 22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
  • 20:48 ejegg: re-enabled donation queue consumer and thank you mailer
  • 20:40 ejegg: updated civicrm from 820bd98 to f9c373e
  • 20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
  • 19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
  • 19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
  • 19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
  • 18:11 ejegg: updated fundraising tools from 9554229 to 8d39806
  • 18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
  • 17:38 bblack: cp1074 - restart varnish backend, mailbox lag
  • 17:24 demon@tin: Started scap: wmf.17 bootstrap
  • 17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
  • 16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
  • 16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
  • 16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
  • 16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
  • 16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
  • 16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
  • 16:08 akosiaris: reboot all kubernetes boxes T170119
  • 15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
  • 15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
  • 15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
  • 15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
  • 14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
  • 14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
  • 14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
  • 14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
  • 14:15 zeljkof: EU SWAT finished
  • 14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
  • 13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
  • 13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
  • 13:28 godog: reimage restbase2005 - T169939
  • 13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
  • 13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
  • 12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
  • 12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102) (duration: 00m 46s)
  • 11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
  • 11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 45s)
  • 11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 46s)
  • 10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
  • 10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
  • 10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
  • 10:11 ema: reboot cp4024 T174891
  • 10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
  • 09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
  • 09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
  • 08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705)
  • 08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705)
  • 07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
  • 07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
  • 06:32 marostegui: Deploy alter table on db1083 - T174509
  • 06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
  • 06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)

2017-09-04

  • 23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973)
  • 22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
  • 17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
  • 17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
  • 17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
  • 16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
  • 16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
  • 16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
  • 16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
  • 15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
  • 14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
  • 14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
  • 13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
  • 13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
  • 13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
  • 12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
  • 12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
  • 12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
  • 11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
  • 11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
  • 11:45 godog: restart varnish on cp1099 - mailbox lag
  • 10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
  • 10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
  • 09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
  • 09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
  • 09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
  • 08:52 elukey: restart zookeeper on conf2003 for jvm security updates
  • 08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
  • 08:47 moritzm: installing mercurial security updates
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
  • 08:39 moritzm: installing libgcrypt20 security updates
  • 08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
  • 08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
  • 07:39 moritzm: installing gnupg security updates
  • 07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
  • 07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
  • 04:52 marostegui: Deploy alter table on db1052 - T168661
  • 02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
  • 02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)

2017-09-03

  • 17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
  • 15:57 ema: restart varnish backend on cp1073 (mailbox lag)
  • 09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
  • 09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
  • 09:25 ema: cp4024 is down, powercycled
  • 09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
  • 00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859

2017-09-01

  • 16:18 bblack: restart varnish backend on cp1074 (mailbox lag)
  • {{safesubst:SAL entry|1=15:15 urandom: Restarting Cassandra: restbase-dev100[5-6]-{a,b}}}
  • {{safesubst:SAL entry|1=15:06 urandom: Restarting Cassandra: restbase-dev1004-{a,b}}}
  • 15:00 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s)
  • 14:28 marostegui: Add 150G to /srv partition on labsdb1001
  • 13:42 marostegui: Rename table pr_index on enwiki on db1089 - T174782
  • 13:10 marostegui: Stop MySQL on db1026 as it will be decommissioned - T174763
  • 12:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
  • 12:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
  • 11:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)
  • 11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)
  • 10:35 elukey: stop puppet on thorium and disable root rsyncs - T174756
  • 10:31 marostegui: Upgrade MariaDB to 10.0.32 on db1081 - T168661
  • 10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)
  • 09:10 godog: bounce graphite-web on graphite1001, problematic query? using all CPU
  • 09:04 ema: lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 08:59 godog: depool restbase200[135] before reimage - T169939
  • 07:22 elukey: restart apache2 and hue on thorium, Analytics sites down, investigating
  • 07:06 marostegui: Power reset db2044 as it is unresponsive - T174764
  • 03:54 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s)
  • 00:57 tstarling@tin: Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s)
  • 00:13 RainbowSprinkles: gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so

2017-08-31

  • 20:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
  • 19:27 mattflaschen@tin: Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s)
  • 19:17 cmjohnson1: powercycled mw1294
  • 19:05 mattflaschen@tin: Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030
  • 18:53 mattflaschen@tin: Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s)
  • 18:51 mattflaschen@tin: Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s)
  • 17:50 demon@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s)
  • 16:35 ejegg: re-enabled donation queue consumer and thank you mail sender
  • 15:25 ejegg: updated civicrm from f00611d to 820bd98
  • 15:20 ejegg: updated SmashPig from 8f4388d to 8eb98c1
  • 15:19 ejegg: turned off donation queue consumer and thank you job
  • 15:18 elukey: restart zookeeper on conf200[2,3] for jvm security updates
  • 13:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)
  • 13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)
  • 13:07 elukey: restart zookeeper on conf2001 for security updates (canary node)
  • 13:00 moritzm: installing augeas security updates
  • 12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)
  • 12:14 marostegui: Upgrade MariaDB on db1084 - T168661
  • 12:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)
  • 12:05 akosiaris: reimage chlorine and kubernetes1004 T170119
  • 12:04 akosiaris: reimage chlorine and kubernetes1004 T10119
  • 11:58 moritzm: restarting nginx on dataset1001/ms1001 to pick up libxml security update
  • 11:21 elukey: restart apache2 on bohrium for libxml + gnutls security updates
  • 11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)
  • 11:17 moritzm: restarting apache on dbmonitor* to pick up libxml security update
  • 10:27 moritzm: restarting nginx on prometheus* to pick up libxml security update
  • 10:22 moritzm: restarting apache on einsteinium/tegmen (Icinga hosts) to pick up libxml security update
  • 10:14 moritzm: restarting nginx on meitnerium/archiva.wikimedia.org to pick up libxml security update
  • 10:13 moritzm: restarting nginx on debug proxies to pick up libxml security update
  • 10:10 moritzm: restarting apache on hafnium to pick up libxml security update
  • 09:57 addshore: extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php run done, 6326 hashes recalculated, T172987
  • 09:50 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 # Should upsert 6326 rows T172987
  • 09:44 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 --dry-run # T172987
  • 09:43 moritzm: restarting apache on auth* to pick up libxml security update
  • 09:40 moritzm: installing libxml2 security updates
  • 09:10 moritzm: installing experimental hhvm-luasandbox package on mw1261 (canary app server) with backported patch for T171392 / T173705
  • 09:04 hashar: contint1001 upgrading git, restarting git-daemon and zuul-merger - T161086
  • 09:01 godog: test reimage of restbase2001 - T169939
  • 08:57 hashar: contint2001 upgrading git, restarting git-daemon and zuul-merger - T161086
  • 08:45 godog: remove /srv/deployment/grafana on naos and tin, unused - T170881
  • 08:41 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@eqiad
  • 08:40 addshore@tin: Synchronized php-1.30.0-wmf.16/extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php: T172987 Add waitForReplication to RecalculateCognateNormalizedHashes script (duration: 00m 47s)
  • 08:20 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@codfw
  • 08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
  • 08:01 marostegui: Rename reader_feedback, reader_feedback_history, reader_feedback_pages tables on dewiki on db1092 - T174586
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
  • 06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 with low weight - T168661 (duration: 00m 46s)
  • 06:19 marostegui: Upgrade MariaDB to 10.0.32 on db1056 - T168661
  • 06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
  • 06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 31 02:55:12 UTC 2017 (duration 7m 9s)
  • 02:48 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 06m 40s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 10m 02s)

2017-08-30

  • 23:10 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 47s)
  • 23:09 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 48s)
  • 23:05 demon@tin: Synchronized wmf-config/unitConversionConfig.json: swat (duration: 00m 47s)
  • 21:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.16
  • 21:03 thcipriani@tin: Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: ProofreadPage: Avoids a stack overflow T173520 (duration: 00m 47s)
  • 20:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.15
  • 20:22 ppchelko@tin: Finished deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages (duration: 01m 05s)
  • 20:21 ppchelko@tin: Started deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages
  • 20:07 urandom: T169939: Decommission Cassandra: restbase1009-c.eqiad.wmnet
  • 19:09 ppchelko@tin: Finished deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages (duration: 01m 17s)
  • 19:07 ppchelko@tin: Started deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages
  • 19:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.16
  • 19:04 demon@tin: Synchronized php: Symlink swap for group1/wmf.16 (duration: 00m 46s)
  • 18:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/374399/4 (duration: 00m 47s)
  • 17:29 urandom: T169939: Decommission Cassandra: restbase1009-b.eqiad.wmnet
  • 17:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 17:12 ejegg: disabled creating CiviMail records for Thank You emails
  • 16:49 XioNoX: rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366
  • 16:34 XioNoX: start work on the cr2<->asw-d-codfw link - T174366
  • 16:13 XioNoX: lowering vrrp priority for the cr2<->asw-d-codfw link - T174366
  • 15:56 elukey: re-added analytics1055 among the hdfs/yarn worker after maintenance
  • 15:49 bblack: cp1049 - restart varnish backend, mailbox lag
  • 15:43 urandom: T169939: Decommission Cassandra: restbase1009-a.eqiad.wmnet
  • 15:02 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1294.eqiad.wmnet
  • 14:31 moritzm: restarting nginx on sodium to pick up libxml security update
  • 14:07 elukey: restart java daemons on druid100[456] for jvm security updates
  • 14:04 moritzm: restarting squid on install* servers to pick up libxml security update
  • 14:04 moritzm: restarting squid on installurl downloaders to pick up libxml security update
  • 14:02 moritzm: restarting apache on krypton to pick up libxml security update
  • 13:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1053 original weight - T168661 (duration: 00m 46s)
  • 13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Scale A/B test bucket sizes by 10 - T172291 (duration: 00m 46s)
  • 13:50 moritzm: restarting squid on url downloaders to pick up libxml security update
  • 13:34 moritzm: installing libxml2 security updates
  • 13:23 moritzm: installing ffmpeg security upadtes
  • 13:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 46s)
  • 13:19 hashar: European SWAT completed
  • 13:16 gehel: restarting wdqs-blazegraph and wdqs-updater on all wdqs nodes for logback config change - T172710
  • 13:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on cywiki - T173054 (duration: 00m 48s)
  • 12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 45s)
  • 12:51 gehel: restart wdqs-updater on wdqs2001 for config change
  • 12:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 47s)
  • 12:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 with low weight - T168661 (duration: 00m 46s)
  • 12:05 marostegui: Upgrade MariaDB to 10.0.32 on db1053 - T168661
  • 12:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T168661 (duration: 00m 47s)
  • 11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T174265 (duration: 00m 46s)
  • 10:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1059 - T168661 (duration: 00m 47s)
  • 09:07 elukey: restart all jvm daemons on druid100[123] for security updates
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 48s)
  • 08:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 07:47 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1228.eqiad.wmnet
  • 07:47 marostegui: Upgrade MariaDB on db1059 to 10.0.32 - T168661
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 for a MariaDB upgrade - T168661 (duration: 00m 53s)
  • 07:14 moritzm: powering down mw1294 for hardware diagnostics (T1674069
  • 07:12 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1294.eqiad.wmnet
  • 07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1055 - T174265 (duration: 00m 52s)
  • 03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 30 03:16:09 UTC 2017 (duration 7m 19s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 15m 42s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 28s)
  • 02:17 demon@tin: Finished deploy [gerrit/gerrit@f33c63f]: Testing, no-op (duration: 00m 07s)
  • 02:17 demon@tin: Started deploy [gerrit/gerrit@f33c63f]: Testing, no-op
  • 01:27 demon@tin: Synchronized wmf-config/InitialiseSettings.php: remove education program from legalteamwiki (duration: 00m 47s)
  • 00:53 demon@tin: Synchronized wmf-config/flaggedrevs.php: killing FR on testwiki (duration: 00m 46s)
  • 00:52 demon@tin: Synchronized dblists/flaggedrevs.dblist: killing FR on testwiki (duration: 00m 47s)
  • 00:05 cwd: turned dedupe off

2017-08-29

  • 23:50 cmjohnson1: reinstalling mw1228
  • 23:34 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Popups/: swat (duration: 00m 47s)
  • 23:33 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: swat (duration: 00m 50s)
  • 21:33 urandom: T169939: Decommission Cassandra: restbase1008-c.eqiad.wmnet
  • 21:16 andrewbogott: restarting designate-sink on labservices1001
  • 21:01 ejegg: updated DjangoBannerStats from 5963e7c to ce95eab
  • 20:56 ejegg: updated payments-wiki from 58d33b8 to bf97cd2
  • 20:49 godog: bounce varnish on cp1074 / cp1099 / cp1072 - mailbox lag
  • 20:38 ppchelko@tin: Finished deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2 (duration: 01m 17s)
  • 20:37 ppchelko@tin: Started deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2
  • 20:36 ppchelko@tin: Finished deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode (duration: 04m 28s)
  • 20:31 ppchelko@tin: Started deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode
  • 19:47 urandom: T169939: Decommission Cassandra: restbase1008-b.eqiad.wmnet
  • 19:34 ottomata: restarting analytics kafka brokers and all mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/
  • 19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.16
  • 19:08 ppchelko@tin: Finished deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config (duration: 06m 48s)
  • 19:02 ottomata: restarting main-eqiad -> analytics kafka mirror maker processes on analytics kafka brokers, something is not working...
  • 19:01 XenoRyet: updated Smashpig from 98c5516 to 8f4388d
  • 19:01 ppchelko@tin: Started deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config
  • 18:19 robh: attempting firmware upgrade on scs-a8-eqiad
  • 18:17 Pchelolo: depooling restbase1008,1009,1010,2003,2005 while cluster reshaping is going on
  • 18:15 demon@tin: Finished scap: bootstrap wmf.16 (duration: 45m 27s)
  • 18:15 urandom: T169939: Decommission Cassandra: restbase1008-a.eqiad.wmnet
  • 17:30 demon@tin: Started scap: bootstrap wmf.16
  • 17:23 ejegg: restarted donation queue consumer and thank you mailer
  • 17:20 Reedy: Disabled oathauth for KartikMistry on wikitech
  • 16:33 ejegg: updated civicrm from 7dfea0d to f00611d
  • 16:32 ejegg: turned off donation queue consumer and thank you mailer
  • 16:04 ejegg: restarted ingenico recurring charge job
  • 15:54 urandom: T169939: Decommission Cassandra: restbase2005-c.codfw.wmnet
  • 14:51 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from db1047 (archived on HDFS) - T172322
  • 14:33 marostegui: Shutdown db1055 to replace its BBU - T174265
  • 13:12 zeljkof: EU SWAT finished
  • 13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
  • 13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
  • 12:37 elukey: restart kafka daemons on kafka1014 for jvm security updates
  • 12:31 godog: bounce pybal on lvs1003 to pick up config changes
  • 11:49 elukey: restart kafka daemons on kafka1013 for jvm security updates
  • 11:43 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1168.eqiad.wmnet
  • 11:30 elukey: restart java daemons on analytics100[1,2] (Hadoop Master nodes) for jvm updates
  • 11:08 marostegui: Stop MariaDB on db1097 to migrate it to file per table - T161088
  • 10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 43s)
  • 10:48 moritzm: installing libxml2 security updates
  • 10:31 moritzm: reimaging mw1168 (video scaler) to jessie
  • 10:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1028 (duration: 00m 42s)
  • 10:15 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1028 (duration: 02m 48s)
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T168661 (duration: 00m 43s)
  • 09:39 marostegui: Update MariaDB on db1064 to 10.0.32 - T168661
  • 09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 for a mariadb upgrade - T168661 (duration: 00m 43s)
  • 09:33 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1169.eqiad.wmnet
  • 09:29 elukey: re-installed pmacct/librdkafka1/kafkacat on rhenium with stretch versions - T173489
  • 09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T168661 (duration: 00m 43s)
  • 08:53 ema: upgrading cache_text to varnish 4.1.8-1wm1
  • 08:45 akosiaris: reprepro copy calico, calico-cni from jessie-wikimedia to stretch-wikimedia (apt.wikimedia.org) T170119
  • 08:43 hashar: Restarting Jenkins for openjdk update
  • 08:42 elukey: restart yarn/hdfs daemons for openjdk security updates
  • 08:42 akosiaris: upload kubernetes_1.7.4-1 to apt.wikimedia.org/stretch-wikimedia/main T170119
  • 08:42 moritzm: restarting archiva to pick up openjdk security update
  • 08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight - T168661 (duration: 00m 43s)
  • 08:35 gehel: restart wdqs-updater on wdqs2001
  • 08:32 hashar: apt-get upgrade on contint1001 and contint2001
  • 08:20 moritzm: reimaging mw1169 (video scaler) to jessie
  • 08:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight - T168661 (duration: 00m 43s)
  • 08:06 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from dbstore1002 to free space (table archived on HDFS) - T172322 T168303
  • 07:41 marostegui: Upgrade MariaDB on db1091 to 10.0.32 - T168661
  • 07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 42s)
  • 06:56 moritzm: installing ghostscript security updates on trusty (Debian already fixed)
  • 06:09 demon@tin: Synchronized scap/plugins/clean.py: no-op, for consistency (duration: 00m 43s)
  • 06:05 marostegui: Restart MariaDB on db1102 and db1095 to pick up new replication filters - T174385
  • 05:49 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 (duration: 02m 50s)
  • 02:34 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 29 02:34:09 UTC 2017 (duration 6m 44s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 00s)
  • 00:25 urandom: T169939: Decommissioning restbase1010-a.eqiad.wmnet

2017-08-28

  • 23:37 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s)
  • 23:31 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s)
  • 23:27 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s)
  • 23:23 reedy@tin: Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s)
  • 23:22 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s)
  • 23:13 cwd: re-enabled dedupe jobs
  • 23:13 ebernhardson@tin: Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)
  • 23:08 ebernhardson@tin: Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
  • 23:07 ebernhardson@tin: Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
  • 22:56 ejegg: turned fundraising orphan rectifier back on
  • 22:46 urandom: T169939: Decommissioning restbase1010-b.eqiad.wmnet
  • 21:48 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s)
  • 21:37 reedy@tin: Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s)
  • 21:06 ejegg: switched nightly Ingenico audit from wr1 to wx file parsing
  • 21:06 reedy@tin: Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s)
  • 21:01 reedy@tin: Synchronized wmf-config/CommonSettings.php: Move email blacklist to meta:Email_blacklist (duration: 00m 45s)
  • 20:53 XioNoX: pushing "aggregate defaults discard" to all the cr* routers - T174364
  • 20:52 urandom: T169939: Decommissioning restbase1010-c.eqiad.wmnet
  • 20:16 ayounsi@tin: Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s)
  • 20:16 ayounsi@tin: Started deploy [librenms/librenms@5fea59c]: (no justification provided)
  • 20:16 XioNoX: upgrading librenms to 1.31.01
  • 20:08 bd808@tin: Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s)
  • 20:07 bd808@tin: Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes)
  • 19:24 ejegg: updated CiviCRM from 200f34b to 7dfea0d
  • 19:24 ottomata: restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s)
  • 19:15 herron: scb1001: restarted pdfrender service
  • 19:14 ottomata: restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:09 ottomata: rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:03 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s)
  • 18:48 gwicke: restarted pdfrender instances in eqiad (T159922)
  • 18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s)
  • 18:47 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s)
  • 18:47 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
  • 18:43 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s)
  • 18:37 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s)
  • 18:33 XioNoX: upgrading librenms to 1.31
  • 18:32 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s)
  • 18:32 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
  • 18:30 ejegg: updated CiviCRM from c30ec13 to 200f34b
  • 18:21 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s)
  • 18:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s)
  • 18:14 niharika29@tin: Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s)
  • 17:59 XioNoX: pushing "aggregate defaults discard" to *ams - T174364
  • 17:56 urandom: T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet
  • 17:31 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
  • 17:19 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
  • 17:11 XioNoX: pushing "aggregate defaults discard" to cr2-knams - T174364
  • 17:09 gehel@tin: Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s)
  • 17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
  • 17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
  • 16:46 demon@tin: Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s)
  • 16:39 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s)
  • 14:39 elukey: restart eventlogging_sync on dbstore1002 - issue after drop of old table
  • 14:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s)
  • 14:31 elukey: drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720
  • 14:28 zeljkof: EU SWAT finished
  • 14:28 zfilipin@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710) (duration: 00m 45s)
  • 14:01 zeljkof: extending EU SWAT for 10 or so minutes to deploy two more patches
  • 13:55 elukey: restart kafka* daemons on kafka1012 for openjdk security updates (canary)
  • 13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Log WikibaseQualityConstraints (T171281) (duration: 00m 44s)
  • 13:42 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872) (duration: 00m 44s)
  • 13:36 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:33 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to grant/remove transwiki user group in dtywiki (T174226) (duration: 00m 44s)
  • 13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152) (duration: 00m 45s)
  • 13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Remove non-transparent background from dty.wiki logos (T174098) (duration: 00m 45s)
  • 12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
  • 12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet
  • 12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
  • 12:07 gehel: restarting nginx for upgrade on wdqs*
  • 12:05 gehel: restarting nginx for upgrade on elastic* / relforge*
  • 12:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s)
  • 11:05 moritzm: installing libxml2 security updates
  • 08:41 moritzm: restarting squid3 on install1002
  • 08:16 marostegui: Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661
  • 08:10 marostegui: Stop MySQL on db1045 - T172679
  • 08:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s)
  • 07:51 moritzm: reimaging mw1259 to jessie (T145742)
  • 07:22 marostegui: Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265
  • 07:12 moritzm: installing openjdk security updates on notebook* hosts
  • 07:05 moritzm: installing openjdk security updates on meitnerium
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s)

2017-08-27

  • 20:35 ariel@tin: Finished deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete (duration: 00m 02s)
  • 20:34 ariel@tin: Started deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete
  • 15:46 akosiaris: upload kubernetes_1.4.6-7 to apt.wikimedia.org/jessie-wikimedia/main T170346
  • 05:30 marostegui: Force BBU relearn on db1055 - T174265
  • 00:45 jynus: correction last log s/db1051/db1055/
  • 00:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051, hw issues, may get lag (duration: 00m 44s)

2017-08-26

  • 15:46 godog: bounce varnish on cp1073 - mailbox lag

2017-08-25

  • 19:09 gehel: actually not killing the "stuck discovery report-updater process on stat1005", it is already gone - T174110
  • 19:07 gehel: kill stuck discovery report-updater process on stat1005 - T174110
  • 16:03 moritzm: created cn=ciadmin group in LDAP to be used for privileged Jenkins access, initial members in ticket (T169557)
  • 14:29 godog: bounce varnish on cp1049 / cp1072 - mailbox lag
  • 14:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db1097 - T168661 (duration: 00m 44s)
  • 13:53 marostegui: Upgrade MariaDB on db1097 to 10.0.32 - T168661
  • 13:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1097 - T168661 (duration: 00m 44s)
  • 13:40 marostegui: Upgrade MariaDB to 10.0.32 on db2019 - T168661
  • 13:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T168661 (duration: 00m 45s)
  • 13:24 marostegui: Upgrade MariaDB on db2037 to 10.0.32 - T168661
  • 13:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 - T168661 (duration: 00m 44s)
  • 12:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T168661 (duration: 00m 44s)
  • 12:29 marostegui: Upgrade MariaDB on db2044 - T168661
  • 12:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T168661 (duration: 00m 43s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2058 - T168661 (duration: 00m 48s)
  • 11:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 (duration: 00m 47s)
  • 09:19 moritzm: restart thumbor to pick up libxml2 security updates
  • 09:12 godog: reimage restbase2001 to test new partman recipe - T169939
  • 08:41 moritzm: restarting apache/hhvm on canary app servers to pick up libxml security update
  • 08:00 volans: upgrading cumin to v1.0.0 on sarin/neodymium
  • 07:55 gehel: reimaging logstash1006 after change of failed disk - T173679
  • 07:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool db1069 with more weight (duration: 00m 44s)
  • 07:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1033, pool db1069 with low weight (duration: 00m 44s)
  • 06:55 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1033 (duration: 00m 43s)
  • 06:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: formatting fixes (duration: 00m 44s)
  • 06:11 jynus@tin: Synchronized wmf-config/db-codfw.php: repool es2013, formatting fixes (duration: 00m 44s)
  • 06:01 marostegui: Upgrade MariaDB to 10.0.32 on db2058 - T168661
  • 06:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2058 - T168661 (duration: 00m 43s)
  • 05:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T168661 (duration: 00m 44s)
  • 05:18 moritzm: re-enable mw1260 (video scaler) for active job processing
  • 05:16 marostegui: Reboot db2065 to pick up new kernel
  • 05:08 marostegui: Upgrade MariaDB on db2065 - T168661
  • 05:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 for a MariaDB upgrade - T168661 (duration: 00m 44s)
  • 01:12 mattflaschen@tin: Synchronized php-1.30.0-wmf.15/extensions/Flow/modules/engine/components/board/features/flow-board-loadmore.js: T173807: Fix Flow infinite scroll (duration: 00m 45s)

2017-08-24

  • 23:17 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: Disable Ogg Theora video transcodes in default config T172445 (duration: 00m 45s)
  • 22:45 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 [keeping static files] (duration: 02m 01s)
  • 22:24 ejegg: updated civicrm from fa882f2 to c30ec13 (pt-br and ja thank you letters)
  • 21:59 gwicke: restarted pdfrender service instances in eqiad / T159922
  • 21:54 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/373701/1 (duration: 00m 45s)
  • 21:13 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Logging: https://gerrit.wikimedia.org/r/#/c/373691/ (duration: 00m 44s)
  • 20:07 andrewbogott: stopping, starting, restarting etc. nodepool
  • 19:58 urandom: T169939: Decommission Cassandra: restbase2003-c.codfw.wmnet
  • 19:24 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.15
  • 19:14 andrewbogott: restarting rabbitmq-server on labcontrol1001
  • 18:38 ejegg: updated payments-wiki from bd9f730 to 58d33b8
  • 18:38 bblack: varnish backend restart - cp1074 - mailbox lag
  • 18:24 urandom: T169939: Decommission Cassandra: restbase2003-b.codfw.wmnet
  • 17:33 arlolra@tin: Finished deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f (duration: 12m 24s)
  • 17:21 arlolra@tin: Started deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f
  • 17:01 jynus: stopping and cloning db1033 to db1069
  • 16:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give normal weight to db1096 - T172679 (duration: 00m 47s)
  • 16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
  • 16:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
  • 16:34 urandom: T169939: Decommission Cassandra: restbase2003-a.codfw.wmnet
  • 16:24 andrewbogott: apt-get upgrade and updated to MW 1.29.1 on wikitech-static. Rebooting to pick up kernel updates.
  • 15:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some weight to db1096 - T172679 (duration: 00m 47s)
  • 15:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1096 depooled to s5 - T172679 (duration: 00m 47s)
  • 14:44 ladsgroup@tin: Synchronized php-1.30.0-wmf.15/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php: Reduce batch size in WikiPageUpdater (T173710) (duration: 00m 48s)
  • 14:14 phuedx: refreshLinks on bawiki (T173994)
  • 14:13 volans: updated cumin package in our APT to version cumin_1.0.0-1
  • 13:19 phuedx@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T171853: Re-enable Page Previews for enwiki and dewiki on the Beta Cluster (duration: 00m 47s)
  • 13:13 phuedx@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
  • 13:12 phuedx@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 49s)
  • 12:45 gehel: killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
  • 12:30 marostegui: Stop MySQL on db1026 to clone db1096 from it - T172679
  • 12:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 to clone db1096 from it T172679 (duration: 00m 47s)
  • 11:36 jynus: disable puppet on db1069 in preparation for reimage
  • 11:08 marostegui: Global rename Papa1234 → Karl-Heinz Jansen - T173859
  • 09:56 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with full weight (duration: 00m 46s)
  • 09:33 volans: executing https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed to get rid of failed ones
  • 08:54 jynus: 16 minute test run of rebuildTermSqlIndex.php on terbium
  • 08:52 marostegui: Drop tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki - T173590
  • 07:24 Amir1: starting the run for rebuildTermIndex (T171460)
  • 02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 24 02:50:40 UTC 2017 (duration 7m 4s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 05m 56s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 09m 50s)
  • 01:46 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue: Deploy 752580637 (T173710) (duration: 00m 49s)

2017-08-23

  • 23:49 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
  • 23:47 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
  • 23:42 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 47s)
  • 23:41 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 49s)
  • 23:28 maxsem@tin: Synchronized php-1.30.0-wmf.15/resources/: https://gerrit.wikimedia.org/r/#/c/373391/ (duration: 00m 48s)
  • 23:27 maxsem@tin: Synchronized wmf-config/MetaContactPages.php: https://gerrit.wikimedia.org/r/#/c/373369/ (duration: 00m 48s)
  • 20:59 urandom: T169939: Truncating MCS tables
  • 20:26 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.15 (duration: 00m 46s)
  • 20:25 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.15
  • 20:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with low weight (duration: 00m 47s)
  • 20:12 thcipriani@tin: Finished scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520 (duration: 04m 11s)
  • 20:08 thcipriani@tin: Started scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520
  • 19:49 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change: pagePreviews: Enable A/B test (BC-only (duration: 00m 47s)
  • 19:45 joal@tin: Finished deploy [analytics/refinery@f467ce1]: Bug fixing deploy (duration: 03m 16s)
  • 19:44 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend: Verify the existence of `url` key when parsing lang objects T172316 (duration: 00m 56s)
  • 19:42 joal@tin: Started deploy [analytics/refinery@f467ce1]: Bug fixing deploy
  • 19:36 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler: Enable WebM playback via ogv.js T172444 (duration: 00m 50s)
  • 19:22 kaldari: foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
  • 19:19 kaldari: mwscript sql.php --wiki=testwiki /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
  • 19:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Remove CitThisPage from blacklist for page previews T173865 (duration: 00m 47s)
  • 19:13 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 49s)
  • 19:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 50s)
  • 19:00 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend/: Verify the existence of key when parsing lang objects https://gerrit.wikimedia.org/r/#/c/373292/ (duration: 00m 49s)
  • 18:57 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Log channel for LoginNotify T173888 (duration: 00m 47s)
  • 18:55 ppchelko@tin: Finished deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains (duration: 01m 33s)
  • 18:54 ppchelko@tin: Started deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains
  • 18:52 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Log everything T173888 (duration: 00m 48s)
  • 18:43 gehel: manually running report updater for discovery golden data on stat1005 - T173333
  • 18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
  • 18:40 gehel: resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
  • 18:39 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
  • 18:32 thcipriani: jenkins restarted, zuul should pick-up queue
  • 18:26 thcipriani: jenkins appears to be having some issues, trying to dump threads now and then restart to see if it helps
  • 18:13 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Allow crats to add people to accountcreator group on mw.or https://gerrit.wikimedia.org/r/#/c/373317/ (duration: 00m 49s)
  • 17:40 bblack: reboot cp4021
  • 16:38 cwd: disabled all dedupe
  • 16:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 [keeping static files] (duration: 01m 32s)
  • 16:24 bd808@tin: Finished deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845) (duration: 00m 39s)
  • 16:23 bd808@tin: Started deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845)
  • 16:22 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 (duration: 03m 44s)
  • 15:35 legoktm: added addshore to "integration" gerrit group (T173233)
  • 15:10 gehel: shutdown logstash1006 for disk replacement - T173679
  • 14:49 moritzm: installing texlive security updates on trusty (Debian already fixed)
  • 14:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4027.ulsfo.wmnet
  • 14:34 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.ulsfo.wmnet
  • 13:31 marostegui: Stop MySQL on db1041 to get it ready for decommission - T173915
  • 13:13 zeljkof: EU SWAT finished
  • 13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow as a Beta feature on wawiki and wawikionary (T172947) (duration: 00m 48s)
  • 12:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
  • 12:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
  • 10:39 jynus: stopping and deleting s1 and s4 from dbstore2001
  • 10:06 joal@tin: Finished deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last) (duration: 03m 30s)
  • 10:03 moritzm: upgrading remaining image scalers to luasandbox 2.0.13
  • 10:02 joal@tin: Started deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last)
  • 09:36 godog: kill older and running tasks from gerrit queue, it was stuck
  • 09:10 moritzm: upgrading remaining API servers in to luasandbox 2.0.13
  • 09:08 godog: upload prometheus-blackbox-exporter 0.7.0+ds1-1~wmf1 to jessie-wikimedia, backported - T169860
  • 08:35 marostegui: Stop Upgrade MySQL on db2073 to 10.0.32 - T168661
  • 08:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T168661 (duration: 00m 59s)
  • 08:18 moritzm: upgrading remaining job runners to luasandbox 2.0.13
  • 07:12 marostegui: Global rename: Opdire657 → Sakiv - T173834
  • 06:55 moritzm: upgrading remaining app servers in to luasandbox 2.0.13
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 23 03:15:36 UTC 2017 (duration 7m 6s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 15m 11s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 03s)

2017-08-22

  • 23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/Linter/includes/LintErrorsPager.php: SWAT: Fix up 11f4a97ba6bcd0c1de (duration: 00m 49s)
  • 23:35 thcipriani@tin: Finished scap: SWAT: Fix typo in the notification message (duration: 21m 48s)
  • 23:13 thcipriani@tin: Started scap: SWAT: Fix typo in the notification message
  • 23:10 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: Remove aborting of BeforePageDisplay hook T173411 (duration: 00m 49s)
  • 21:18 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Retry CodeMirror deployment T170966 (duration: 00m 49s)
  • 20:41 ejegg: updated CiviCRM from 4aa177b to fa882f2
  • 20:25 mutante: restbase-dev* - puppet runs fail due to E: Version '3.11.0' for 'cassandra' was not found
  • 20:00 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.15
  • 19:56 urandom: starting cassandra restbase2004-a and restbase2006-c, OOMs
  • 19:46 thcipriani@tin: Finished scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache (duration: 45m 55s)
  • 19:00 thcipriani@tin: Started scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache
  • 18:50 robh: labservies1002 update completed (not 1003, typo)
  • 18:50 robh: labservies1003 update completed
  • 18:40 robh: firmware update of labservices1002 in progress
  • 17:54 andrewbogott: removing obsolete apache2 and puppetmaster packages from labcontrol boxes for https://phabricator.wikimedia.org/T171786
  • 17:07 thcipriani: starting branch cut for 1.30.0-wmf.15
  • 13:58 godog: bounce varnish on cp1074 / cp1049 / cp1073 - mailbox problems
  • 13:48 zeljkof: EU SWAT finished
  • 13:45 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
  • 13:44 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
  • 13:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseWikimediaShopLink to true for ptwiki (T173768) (duration: 00m 45s)
  • 10:41 moritzm: upgrading hhvm-luasandbox on mw1293-mw1295 (image scalers)
  • 10:21 Amir1: another run of rebuildTermSqlIndex (T171460)
  • 09:54 moritzm: upgrading hhvm-luasandbox on mw1189-mw1208 (API servers)
  • 09:42 marostegui: Renaming user Darwinius → DarwIn - T173159
  • 09:35 moritzm: upgrading hhvm-luasandbox on mw1180-1188 and mw1209-mw1220 (app servers)
  • 09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T172996 (duration: 00m 44s)
  • 08:59 akosiaris: upload apertium-bel-rus_0.2.0~r81186-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:55 moritzm: upgrading hhvm-luasandbox on mw1161-mw1167 (job runners)
  • 08:43 akosiaris: upload apertium-rus_0.1.0~r81184-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:43 akosiaris: upload apertium-bel_0.1.0~r81357-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:35 godog: bounce varnish on cp1072 - mailbox problems
  • 08:33 godog: bounce varnish on cp4015 - mailbox problems
  • 08:24 marostegui: Drop s4 from db1095 with NO replication - T172996
  • 08:23 moritzm: upgrading hhvm-luasandbox on deployment servers / script runners
  • 07:34 marostegui: Stop replication on db1064 sanitarium2 and sanitarium3 master to move labsdb1009,10 and 11 s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T172996 (duration: 00m 52s)
  • 07:28 moritzm: installing ruby security updates on trusty (Debian already fixed)
  • 07:11 moritzm: installing c-ares security updates on trusty (Debian already fixed)
  • 06:54 moritzm: installing graphite2 (the image library, not the metrics tool) security updates on trusty (Debian already fixed)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 22 02:30:47 UTC 2017 (duration 6m 55s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 07m 11s)

2017-08-21

  • 22:50 robh: rebooting tin for firmware update since its idle
  • 22:20 mutante: praseodymium - installing BIOS upgrade, reboot
  • 22:11 mutante: xenon - installing BIOS upgrade
  • 21:58 mutante: cerium (cassandra test) - rebooting for firmware upgrade
  • 21:54 mutante: cerium - installing Dell BIOS upgrade (T162850)
  • 21:26 mutante: bast2001 - running Dell BIOS firmware upgrade (T162850)
  • 21:20 arlolra@tin: Finished deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b (duration: 10m 59s)
  • 21:09 arlolra@tin: Started deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b
  • 18:40 niharika29@tin: Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)
  • 18:34 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
  • 18:33 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
  • 18:30 niharika29@tin: Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)
  • 18:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Increase AbuseFilter autodisable thresholds for Meta-Wiki T173633 (duration: 00m 44s)
  • 18:22 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add CookBook and Cookbook Talk NS on hiwikibooks T173398 (duration: 00m 45s)
  • 18:16 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Administrators to add/remove 'transwiki' at nowiktionary T172365 (duration: 00m 45s)
  • 18:12 niharika29@tin: Synchronized static/images/: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
  • 18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
  • 17:22 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038 (duration: 00m 14s)
  • 17:22 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038
  • 17:12 gehel@tin: Finished deploy [wdqs/wdqs@a1c4f1f]: (no justification provided) (duration: 02m 27s)
  • 17:10 gehel@tin: Started deploy [wdqs/wdqs@a1c4f1f]: (no justification provided)
  • 15:40 bblack: cp1099 - varnish backend restart for mailbox lag
  • 14:55 mobrovac: cxserver depool scb2001 to debug failed checks - T173038
  • 14:54 mobrovac@tin: Finished deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038 (duration: 00m 18s)
  • 14:53 mobrovac@tin: Started deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038
  • 14:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T163190 (duration: 00m 44s)
  • 14:04 zeljkof: EU SWAT finished
  • 14:04 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
  • 14:02 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
  • 13:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 49s)
  • 13:52 jynus: stop dbstore2001 mariadb@s4, start mariadb@s7
  • 13:52 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 44s)
  • 13:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set X-Frame-Options: SAMEORIGIN if UploadWizard enabled (T173631) (duration: 00m 44s)
  • 13:46 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990
  • 13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 44s)
  • 13:36 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 45s)
  • 13:27 jynus: stop dbstore2001 mariadb@s7
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
  • 13:26 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
  • 13:26 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990
  • 13:11 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
  • 13:10 zfilipin@tin: Synchronized dblists/closed.dblist: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
  • 12:52 marostegui: Stop replication on db1079 and db1041 to compare their data - T163190
  • 12:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T163190 (duration: 00m 45s)
  • 12:35 marostegui: Stop MySQL on db1015 to decommission it - T173570
  • 09:50 jynus: restart dbstore2001 mariadb@s1
  • 09:40 jynus: restart dbstore2001 mariadb@x1
  • 09:29 moritzm: upgrading hhvm-luasandbox on mw1262-1265
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)
  • 09:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)
  • 09:01 moritzm: upgrading hhvm-luasandbox on mw1261
  • 08:08 marostegui: Rename tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki on db1078 - T173590
  • 07:42 marostegui: Drop cx_drafts table from x1 - T172364
  • 07:13 marostegui: Stop s4 replication thread on db1095 - T172996
  • 07:09 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 (duration: 00m 45s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 21 02:36:58 UTC 2017 (duration 7m 3s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 57s)

2017-08-19

  • 13:17 Amir1: another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井 俊光 → Toshimit) (T173419)
  • 13:00 Amir1: running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
  • 12:47 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
  • 03:21 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on test.wikidata and mediawiki.org - Id4d42d8c53 (duration: 00m 45s)

2017-08-18

  • 21:05 mutante: gerrit2001 - restarted gerrit again after reverting gerrit:372426, using systemctl commands, not 'service' or init.d
  • 20:55 mutante: restarting gerrit on gerrit2001
  • 20:48 demon@tin: Synchronized scap/plugins/clean.py: Completeness (duration: 00m 44s)
  • 20:43 RainbowSprinkles: prior thing was a no-op, testing
  • 20:42 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 43s)
  • 19:03 bblack: cp1074 - varnish backend restart (mailbox lag)
  • 19:01 bblack: reboot cp4021-8
  • 18:21 bblack: puppeting cp4021-8 - expect possibility of ipsec alerts, etc...
  • 13:32 Amir1: another run of /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item --from-id $(tail -100 /tmp/rebuildTermSqlIndex.log | grep -E "Processed up to page (\d+?)" | sed -E "s/Processed up to page //; s/ \(Q.+?//" | tail -1) >>/tmp/rebuildTermSqlIndex.log 2>&1
  • 13:26 Amir1: ladsgroup@terbium:~$ mwscript namespaceDupes.php --wiki=hiwikiversity (T172977)
  • 12:44 Amir1: start of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
  • 11:36 jynus: change topology of dbstore2001:x1 and dbstore2002:x1
  • 11:00 elukey: reboot dbstore1002 for kernel updates
  • 10:34 elukey: restart mysql on dbstore1002 - attempt to reclaim space after big table drop (stop slaves and el_sync, check running queries, stop mysql, check process, start mysql)
  • 09:55 Amir1: one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)
  • 09:46 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)
  • 09:42 moritzm: installing libmspack security updates
  • 09:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2077 - T168409 (duration: 00m 44s)
  • 07:28 marostegui: Stop MySQL on db2077 to copy it to dbstore2001 - T168409
  • 07:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2077 - T168409 (duration: 00m 44s)
  • 07:20 moritzm: installing openjdk-7 security updates on trusty
  • 06:58 jynus: and nginx
  • 06:55 jynus: systemctl restart squid3.service on install2002, apt seem stuck on some servers
  • 06:24 jynus: stopping and upgrading db2033
  • 05:43 jynus: upgrading and restarting all mariadb instances on dbstore2002
  • afk: updated fundraising dash from 696a3ff to 7dfc969
  • 00:48 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (again) to 1k per bucket per day (duration: 00m 44s)
  • 00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (duration: 00m 44s)

2017-08-17

  • 23:31 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Revert "Disable cirrus MLR ab test" (duration: 00m 44s)
  • 23:21 thcipriani@tin: Synchronized php-1.30.0-wmf.14/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FilterGroup.js: SWAT: RCFilters: Fix validation for single_option groups T173303 (duration: 00m 44s)
  • 23:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reapply "Remove temporary wgStructuredChangeFiltersEnableExperimentalViews setting" (duration: 00m 45s)
  • 22:10 ppchelko@tin: Finished deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping (duration: 01m 14s)
  • 22:09 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/specials/SpecialNewpages.php: Restore the newFromId() approach in SpecialNewpages::feedItemDesc T173541 (duration: 00m 46s)
  • 22:09 ppchelko@tin: Started deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping
  • 22:07 gwicke: restarting pdfrender on scb100* nodes
  • 21:38 ejegg: updated dash from bec0077 to 696a3ff
  • 21:28 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify on sites without Echo too. :|https://gerrit.wikimedia.org/r/#/c/372456/ (duration: 00m 44s)
  • 21:19 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify finally! Yippeeeeee https://gerrit.wikimedia.org/r/#/c/372450/ (duration: 00m 45s)
  • 21:17 bblack: cp1063: varnish backend restart (mailbox lag)
  • 20:07 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.14
  • 19:28 chasemp: disable puppet for cloud things for some careful refactor merging
  • 19:14 thcipriani@tin: Synchronized php: group1 wikis to wmf.14 (duration: 00m 46s)
  • 19:12 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to wmf.14
  • 19:07 thcipriani@tin: Finished scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520 (duration: 06m 13s)
  • 19:01 thcipriani@tin: Started scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520
  • 18:28 ebernhardson: restart logstash on logstash1003 to see why its not reading EventError messages from kafka
  • 18:24 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/RevisionSlider/: Revert Reintroduce hover and bar clicking - https://gerrit.wikimedia.org/r/#/c/372384/ (duration: 00m 48s)
  • 18:16 jynus: upgrading and restarting all mariadb instances on dbstore2001
  • 18:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: Log usage statistics https://gerrit.wikimedia.org/r/#/c/372214/ (duration: 00m 51s)
  • 18:03 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/UploadWizard/UploadWizard.config.php: SWAT: Preserve array keys (language keys) when sorting the language dropdown T173522 (duration: 00m 51s)
  • 17:53 jynus: restarting mariadb (dbstore2001:x1) to test new buffer pool configuration
  • 17:20 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520
  • 17:10 RainbowSprinkles: gerrit: cpu spikes, lots of large gc logs. Looking into it. (nb: things might be a little slow, but it is /up/)
  • 16:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173
  • 16:51 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: Avoid lock acquisition errors for multi-title refreshlinks jobs T173462 (duration: 00m 51s)
  • 16:45 ejegg: updated SmashPig from c501f53 to 98c5516
  • 16:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2077 - T170662 (duration: 00m 49s)
  • 16:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2077 - T170662 (duration: 00m 51s)
  • 16:16 urandom: T169939: RESTBase: converting (33) new keyspaces to time-windowed compaction
  • 16:11 urandom: T169939: Lower eqiad compaction throughput from 20MB/s to 15MB/s
  • 15:46 godog: reimage restbase2001 - T169939
  • 15:42 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
  • 14:32 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3036.*
  • 13:30 zeljkof: EU SWAT finished
  • 13:24 zfilipin@tin: Synchronized static/favicon/wikiversity.ico: SWAT: Update wikiversity favicon (T160491) (duration: 00m 50s)
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod to any for srwikiquote (T172974) (duration: 00m 51s)
  • 13:08 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Add one throttling exception (T173444) (duration: 00m 51s)
  • 12:38 moritzm: installing libgd/libsoup security updates
  • 12:08 urandom: T169939: Decommissioning Cassandra/restbase2001-c.codfw.wmnet
  • 12:07 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet
  • 10:29 gilles: Thumbor stress test finished
  • 10:20 gilles: Load testing Thumbor
  • 10:13 godog: roll-restart nginx on thumbor to apply https://gerrit.wikimedia.org/r/372199
  • 08:36 godog: roll-restart thumbor to apply webp support change
  • 08:30 godog: restart varnish on cp1062 to fix mailbox lag
  • 08:05 godog: reboot ms-be2021, unreachable on network
  • 06:24 marostegui: Stop slave on db2047 to fix duplicate keys - T151029
  • 05:55 marostegui: Stop replication on db2077 to fix duplicate entries - T151029
  • 05:48 marostegui: Stop replication in sync on db1078 and db1015 - T164488
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 17 03:11:13 UTC 2017 (duration 7m 18s)
  • 03:03 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 03m 58s)
  • 02:50 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 33s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 52s)
  • 00:45 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet

2017-08-16

  • 22:30 urandom: T169939: Decommissioning Cassandra/restbase2001-a.codfw.wmnet
  • 22:14 urandom: T169939: Cleaning up wikipedia parsoid snapshots
  • 22:13 urandom: T169939: Rolling restart of Cassandra complete
  • 21:38 bawolff: deleting private info from securepoll_votes that the script missed due to ref-integ issues for old elections (T173393)
  • 21:37 thcipriani: train is on hold pending resolution of T173462
  • 21:33 bawolff: deleting private info in enwiki arbcom1_vote table (T173393)
  • 21:30 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack d
  • 21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462
  • 21:21 thcipriani@tin: Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)
  • 20:46 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack b
  • 20:45 arlolra@tin: Finished deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e (duration: 08m 52s)
  • 20:37 arlolra@tin: Started deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e
  • 19:57 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack a
  • 19:52 MaxSem: Manually cleaning up PI on enwiki (T173393)
  • 19:37 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.14 (duration: 00m 46s)
  • 19:35 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.14
  • 19:06 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack d
  • 18:57 thcipriani@tin: Synchronized wmf-config/CirrusSearch-common.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART II (duration: 00m 50s)
  • 18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART I (duration: 00m 51s)
  • 18:41 thcipriani@tin: Synchronized wmf-config: SWAT: Apply token count limits to phrase queries on all wikis T172653 (duration: 00m 53s)
  • 18:30 thcipriani@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 50s)
  • 18:29 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 51s)
  • 18:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable group1 T163380 (duration: 00m 54s)
  • 18:18 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack c
  • 17:31 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack b
  • 17:07 demon@tin: Synchronized php-1.30.0-wmf.14/extensions/AntiSpoof/SpoofUser.php: T173394 (duration: 00m 51s)
  • 15:54 marostegui: Stop MySQL and shutdown db1078 for HW checks - T173365
  • 15:43 elukey: drop PageContentSaveComplete_5588433_15423246 from db1047 and dbstore1002 (analytics-slaves)
  • 15:41 godog: delete outdated CFs cassandra metrics from graphite2002 and graphite1003
  • 15:35 marostegui: Rename cx_drafts table on db1029 - T172364
  • 15:09 godog: restart varnish on cp1072 to clear mailbox lag
  • 14:56 godog: restart varnish on cp1099 to clear mailbox lag
  • 14:48 godog: restart varnish on cp1074 to clear mailbox lag
  • 14:44 godog: restart varnish on cp1049 to clear mailbox lag
  • 14:38 gilles: Thumbor stress test finished
  • 14:29 gilles: Stress testing Thumbor from single IP
  • 13:57 zeljkof: EU SWAT finished
  • 13:35 zfilipin@tin: Synchronized dblists/pp_stage1.dblist: SWAT: pagePreviews: Deploy to next 100 stage 1 wikis (T162672) (duration: 00m 50s)
  • 13:09 zfilipin@tin: Synchronized portals: (no justification provided) (duration: 00m 52s)
  • 13:08 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 52s)
  • 12:25 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)
  • 12:25 moritzm: powercycling cp3036
  • 12:24 marostegui: Compressing InnoDB on db2077 - T168409
  • 12:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: cp3036.esams.wmnet
  • 10:46 marostegui: Stop replication in sync on db1015 and db1078 - T164488
  • 10:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2076 - T170662 (duration: 00m 51s)
  • 10:03 godog: copy ubuntu-font-family-sources to stretch-wikimedia - T170817
  • 08:37 marostegui: Drop wikigrok tables from s1, s3 and s5 - T172020
  • 08:12 marostegui: Stop MySQL on db2047 to copy its content to db2077 - T170662
  • 08:10 moritzm: bounced pdfrender on scb1004 (T159922)
  • 08:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T170662 (duration: 01m 06s)
  • 07:25 marostegui: Stop MySQL on db2076 to copy its content to dbstore2001 - T168409
  • 07:08 elukey: executed sudo find -type f -mtime +30 -exec rm {} \; in /var/log/carbon to free some space
  • 06:40 marostegui: Run pt-table-checksum on s3 for revision table - T164488
  • 06:01 marostegui: Stop replication on db2076 to fix duplicate entries - T151029
  • 03:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 16 03:35:43 UTC 2017 (duration 7m 21s)
  • 03:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 16m 32s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 08s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 51s)
  • 00:43 dereckson@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372088) (duration: 00m 51s)
  • 00:42 dereckson@tin: Synchronized php-1.30.0-wmf.14/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372051) (duration: 00m 51s)
  • 00:38 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on three French wikis (T154371) + Fixes for Wikidata: Remove wbq_evaluation logging, Update Wikidata property blacklist (Gerrit:367913 and Gerrit:370846) (duration: 00m 53s)
  • 00:02 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on test wikis (T173388) (duration: 00m 51s)

2017-08-15

  • 23:58 dereckson@tin: Synchronized php-1.30.0-wmf.14/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 50s)
  • 23:55 dereckson@tin: Synchronized php-1.30.0-wmf.13/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 54s)
  • 23:14 Dereckson: Ran updateArticleCount.php on sr.wikiquote (T172974)
  • 23:12 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for hiwikiversity (T172977) (duration: 00m 53s)
  • 22:50 ppchelko@tin: Started restart [changeprop/deploy@444223d]: (no justification provided)
  • 22:45 Pchelolo: scb - codfw: enabling puppet and starting changeprop back T169939
  • 21:40 mobrovac: scb - codfw: disabling puppet and stopping changeprop to release load on Cassandra while dropping old tables - T169939
  • 21:26 urandom: Starting Cassandra restbase2005-b
  • 19:24 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14
  • 19:19 mobrovac@tin: Finished deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939 (duration: 02m 50s)
  • 19:16 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939
  • 19:14 demon@tin: Finished scap: bootstrap wmf.14 v2.0 (duration: 30m 56s)
  • 19:04 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones) - T169939
  • 19:03 mobrovac@tin: Finished deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939 (duration: 04m 39s)
  • 18:58 mobrovac@tin: Started deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939
  • 18:43 demon@tin: Started scap: bootstrap wmf.14 v2.0
  • 18:39 demon@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3982832257" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 05s)
  • 18:35 demon@tin: Started scap: bootstrap wmf.14
  • 18:18 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 [keeping static files] (duration: 02m 47s)
  • 17:59 bsitzmann@tin: Finished deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362) (duration: 04m 00s)
  • 17:55 bsitzmann@tin: Started deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362)
  • 15:21 mobrovac: restarting pdfrender on scb1001, added some debug messages to help us diagnose T159922
  • 13:42 gehel: cleanup of old leftover logs on deployment-logstash2:/var/log/logstash
  • 13:21 moritzm: bounced pdfrender on scb1001 (T159922)
  • 11:32 moritzm: installing Linux updates on stretch systems (no reboots yet)
  • 10:45 moritzm: installing PHP security updates on trusty
  • 10:09 jynus: rebooting db1078
  • 09:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1078 (duration: 00m 49s)
  • 07:56 moritzm: installing cvs security updates
  • 07:18 elukey: restart pdfrender on scb1003
  • 02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 15 02:56:14 UTC 2017 (duration 6m 42s)
  • 02:49 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 53s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 58s)

2017-08-14

  • 22:16 Dereckson: Previous log entry is related to Gerrit:371960 and Gerrit:371961.
  • 22:15 dereckson@tin: Synchronized php-1.30.0-wmf.11/extensions/EventBus/JobQueueEventBus.php: JobQueueEventBus: Populate the database field + not set properties are accessed (duration: 00m 48s)
  • 20:44 ppchelko@tin: Finished deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation (duration: 09m 35s)
  • 20:35 ppchelko@tin: Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation
  • 17:09 gehel: resetting mode on stat1005:/srv/published-datasets/discovery recursively
  • 16:29 elukey: execute sudo find -type f -mtime +60 -exec rm {} \; in /var/log/carbon on graphite2001 to free some space in /
  • 13:32 elukey: Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681
  • 13:08 zeljkof: nothing for EU SWAT
  • 12:21 jynus: stopping replication on all instances of dbstore2001 T169516
  • 11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s)
  • 11:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s)
  • 10:34 marostegui: Restart MySQL on db1069 to pick up new replcation filters
  • 10:27 marostegui: Restart MySQL on db1095 to pick up new replication filters
  • 09:16 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet
  • 09:07 _joe_: stopping thumbor on thumbor2001 after depooling it for testing
  • 09:05 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s)
  • 02:39 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s)

2017-08-13

  • 22:45 ebernhardson: restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152
  • 19:36 godog: upload python-thumbor-wikimedia 1.2 - T161719
  • 19:01 godog: bounce pdfrender on scb1001 and scb1003 - T159922
  • 18:46 godog: bounce carbon and uwsgi on graphite1003
  • 17:53 reedy@tin: Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s)
  • 17:52 reedy@tin: Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s)
  • 17:51 reedy@tin: Synchronized refresh-dblist: phpcs (duration: 00m 48s)

2017-08-12

  • 20:00 krinkle@tin: Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)
  • 16:09 Reedy: Deleted some bogus user languages from commonswiki.user_properties
  • 15:25 elukey: powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346

2017-08-11

  • 21:47 legoktm@tin: Finished scap: (no justification provided) (duration: 07m 09s)
  • 21:41 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet
  • 21:40 legoktm@tin: Started scap: (no justification provided)
  • 21:40 legoktm@tin: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s)
  • 21:02 ebernhardson: unban elastic1017 from elasticsearch cluster
  • 20:52 bblack: varnish backend restart on cp1099
  • 20:46 legoktm@tin: Started scap: Deploying Timeless (try 2) - T154371
  • 20:42 legoktm@tin: scap aborted: Deploying Timeless - T154371 (duration: 05m 10s)
  • 20:36 legoktm@tin: Started scap: Deploying Timeless - T154371
  • 20:19 bblack: varnish backend restart on cp1049 + cp1074 (mailbox lag)
  • 19:59 ebernhardson: ban elastic1017 from eqiad search cluster
  • 18:28 moritzm: installing subversion security updates
  • 17:19 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s)
  • 17:05 twentyafterfour: Deploying phabricator security update
  • 16:21 jynus_: stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki
  • 15:41 jynus_: stopping db2075 to clone it to dbstore2001
  • 15:39 moritzm: installing git security updates on trusty (jessie/stretch already fixed)
  • 15:38 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
  • 13:51 elukey: moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506
  • 13:49 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s)
  • 13:49 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:42 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s)
  • 13:42 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:25 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s)
  • 13:24 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:16 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
  • 13:15 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:00 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 17s)
  • 13:00 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:51 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 04s)
  • 12:51 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:44 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 05s)
  • 12:44 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T151029 (duration: 00m 48s)
  • 11:25 jynus: stopping and upgrading labsdb1010
  • 11:19 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
  • 09:28 jynus: stopping and restarting es2013 for upgrade
  • 09:07 jynus: stopping and restarting db2046 for upgrade
  • 07:35 elukey: restart pdfrender on scb1004

2017-08-10

  • 23:43 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/371196/2 (duration: 00m 47s)
  • 23:26 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 47s)
  • 23:25 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 46s)
  • 22:31 twentyafterfour@tin: Synchronized php-1.30.0-wmf.13/includes/specials/SpecialDoubleRedirects.php: Hopefully fix T173045 (duration: 00m 48s)
  • 22:17 moritzm: installing git security-updates
  • 20:51 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371 (duration: 00m 21s)
  • 20:51 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371
  • 20:50 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371 (duration: 00m 09s)
  • 20:50 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371
  • 20:35 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371 (duration: 00m 05s)
  • 20:34 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371
  • 20:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371 (duration: 18m 17s)
  • 19:42 mobrovac@tin: Started deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371
  • 19:35 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 47s)
  • 19:33 reedy@tin: Synchronized wmf-config/: Newsletter on testwiki (duration: 00m 49s)
  • 19:31 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340 (duration: 01m 08s)
  • 19:30 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340
  • 19:30 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340 (duration: 00m 34s)
  • 19:29 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340
  • 19:28 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340 (duration: 00m 12s)
  • 19:28 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340
  • 19:27 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: All wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
  • 19:16 godog: restart cassandra instances on xenon to test logstash-logback-encoder deploy
  • 19:13 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340 (duration: 00m 03s)
  • 19:13 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340
  • 18:56 Reedy: created newsletter tables on testwiki
  • 18:55 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/Newsletter/: sql file updates (duration: 00m 52s)
  • 18:53 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaMaintenance/createExtensionTables.php: newsletter (duration: 00m 52s)
  • 18:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Fix commons/commonswiki snafu (duration: 00m 52s)
  • 17:53 kartik@tin: Finished deploy [cxserver/deploy@f43ef96]: (no justification provided) (duration: 00m 44s)
  • 17:52 kartik@tin: Started deploy [cxserver/deploy@f43ef96]: (no justification provided)
  • 17:42 kartik@tin: Finished deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3 (duration: 00m 40s)
  • 17:42 kartik@tin: Started deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3
  • 16:23 ema: cp1072: restart varnish backend
  • 15:48 XioNoX: troubleshoting interface errors between pfw3-codfw and fasw-codfw
  • 15:09 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
  • 15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T151029 (duration: 00m 50s)
  • 15:03 marostegui: Poweroff es2013 for maintenance - T172265
  • 14:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2075 - T170662 (duration: 00m 51s)
  • 14:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T151029 (duration: 00m 51s)
  • 14:11 elukey: restart kafka1012 temporary with some logs to TRACE to debug T172681
  • away: updated civicrm from 200abc2 to 4aa177b
  • 13:43 marostegui: Compress cebwiki on db1095 - T153058
  • 13:32 zeljkof: EU SWAT finished
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod to any on srwikisource (T172974) (duration: 00m 53s)
  • 13:18 zeljkof: EU SWAT, part two
  • 13:17 marostegui: Drop m3 databases from dbstore1002 - T156758
  • 13:16 zeljkof: EU SWAT finished
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on knwiki (T172894) (duration: 00m 52s)
  • 12:56 marostegui: Stop MySQL on db2045 to upgrade socket location - T148507
  • 12:07 elukey: restored varnishakafka on cp3032
  • 11:17 elukey: disabled puppet on cp3032 and restarted varnishkafka with debug logging
  • 09:58 jynus: continuing cloning of dbstore2002 to dbstore2001
  • 09:30 gehel: repooling wdqs2001, long after data reload completed
  • 09:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
  • 09:27 gehel@tin: Finished deploy [wdqs/wdqs@c186e3e]: (no justification provided) (duration: 01m 31s)
  • 09:25 gehel@tin: Started deploy [wdqs/wdqs@c186e3e]: (no justification provided)
  • 09:17 jynus: disabling lag notification on all s4 replicas
  • 08:59 elukey: update librdkafka1 to 0.9.4.1 on eventlog1001
  • 08:15 elukey: add 50G to carbon lv on graphite1003 and 100G on graphite2002
  • 07:32 jynus: enabling semisymc master replication on db1068
  • 07:29 jynus: disabling semisync slave replication on all SSD hosts on s4
  • 06:45 elukey: powercycle mw2256 - T163346
  • 06:38 elukey: restart pdfrender on scb1004
  • 06:04 Dereckson: Removed 2FA for GoldRingChip account (T172878)
  • 03:23 bblack: cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn"
  • 03:20 bblack: batched cumin puppet agent run on all hosts (not forced)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s)

2017-08-09

  • 22:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s)
  • 21:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s)
  • 21:11 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s)
  • 21:05 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s)
  • 21:01 godog: add 100G to carbon lv on graphite1003 and graphite2002
  • 20:58 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
  • 20:54 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s)
  • 20:51 marostegui: Remove m3 replication from dbstore1002 - T156758
  • 20:47 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy
  • 20:43 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s)
  • 20:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s)
  • 20:35 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3
  • 20:22 twentyafterfour: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
  • 18:57 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2
  • 18:39 marostegui: Stop replication on db2045 to fix duplicate keys - T151029
  • 18:36 urandom: Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness)
  • 18:27 urandom: Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness)
  • 18:10 ebernhardson: restart es on elastic1018 to test interleaved numa
  • 17:50 marostegui: Add db2076 to tendril - T170662
  • 17:41 marostegui: Compress innodb on s6 on db2076 - T170662
  • 17:31 legoktm: changed mediawiki/vendor to fast forward only
  • 17:24 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s)
  • 17:23 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
  • 16:35 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s)
  • 16:30 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation
  • 16:29 urandom: T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal)
  • 16:15 marostegui: Stop mysql on db2046 to copy its content to db2076 - T170662
  • 16:14 elukey: rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793
  • 16:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s)
  • 16:08 gehel: config update on logstash, a few logs might be lost during restart - T172713
  • 15:19 XioNoX: removing old pfw related config from cr1-codfw - T171970
  • 14:08 mutante: purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404
  • 14:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s)
  • 14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s)
  • 13:55 marostegui: Reboot db2076 for maintenance - T170662
  • 13:21 marostegui: Stop replication on db2075 for maintenance - T170662
  • 13:18 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s)
  • 13:10 gehel@tin: Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s)
  • 13:09 gehel@tin: Started deploy [wdqs/wdqs@6620e0f]: (no justification provided)
  • 12:41 gehel: restarting updater on wdqs1001 (real fix coming up soon
  • 10:20 jynus: stopping dbstore2002's mysqls and cloning them to dbstore2001
  • 08:39 jynus: disable puppet on dbstore2001, about to be reimaged
  • 07:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s)
  • 07:12 jynus: stopping replication in sync between db1069 and db1065, db1044, db1064, db1070
  • 07:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s)
  • 06:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s)
  • 06:11 jynus: stopping db1069 (s7) and db1079 replication in sync
  • 06:03 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085, depool db1079 (duration: 00m 51s)
  • 05:42 jynus: stopping db1069 (s6) and db1085 replication in sync
  • 05:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060, depool db1085 (duration: 00m 50s)
  • 04:56 jynus: stopping db1069 (s2) and db1060 replication in sync
  • 04:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 58s)
  • 04:48 jynus: restarting pdfrender on scb1001, unresponsive
  • 04:09 jynus: powercycling mw2256
  • 03:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 33s)
  • 02:56 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 45s)
  • 02:38 ebernhardson: restart elasticsearch1017 with niofs store
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 38s)
  • 00:27 ebernhardson: test vm.zone_reclaim_mode=0 on elastic1017

2017-08-08

  • 23:23 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats on WMF wikis to grant and remove 'confirmed' (T101983) (duration: 00m 51s)
  • 21:25 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.13 refs T170631
  • 21:08 ppchelko@tin: Finished deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API (duration: 08m 31s)
  • 21:05 twentyafterfour@tin: Finished scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631 (duration: 42m 55s)
  • 20:59 ppchelko@tin: Started deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API
  • 20:32 twentyafterfour: updated mediawiki.org changelog for 1.30.0-wmf.13
  • 20:22 twentyafterfour@tin: Started scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
  • 19:46 ema: restart varnish backend on cp1074
  • 19:45 godog: bounce thumbor-instances on thumbor1003 to make sure all memory limits are applied
  • 19:44 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="test2wiki" --outdir="/tmp/scap_l10n_224168097" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 07m 33s)
  • 19:36 twentyafterfour@tin: Started scap: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
  • 18:28 XioNoX: frack-codfw moved to new infrastructure
  • 18:27 elukey: re-enabled irc-echo after the puppet shower
  • 18:11 elukey: stop ircecho to avoid puppet shower
  • 16:59 twentyafterfour: Branching mediawiki/master to mediawiki/wmf/1.30.0-wmf.13 refs T170631
  • 16:04 elukey: rolling restart of varnishkafka-webrequest to apply https://gerrit.wikimedia.org/r/#/c/370659/ (puppet automatically restarts)
  • 15:03 XioNoX: starting pfw-codfw migration - T171970
  • 14:19 elukey: restart of all the varnishkafka statsv/eventlogging instances on caching hosts to pick up https://gerrit.wikimedia.org/r/370644 (puppet automatic restarts)
  • 14:16 elukey: set mw2256 pooled=inactive + downtime to allow BIOS upgrade - T163346
  • 14:07 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add copyright info for Wikidata API (T112606) (duration: 00m 47s)
  • 13:56 gehel: dump of task API for elasticsearch eqiad - T169498
  • 13:54 ladsgroup@tin: Synchronized static/images/project-logos: SWAT: Fix srwiki logos (T150618) (duration: 00m 48s)
  • 13:00 elukey: restart varnishkafka-webrequest with kafka.broker.version.fallback=0.9.0.1 + kafka.api.version.request=false on cp3032 (local test, to rollback remove the lines from /etc/varnishkafka/webrequest.conf)
  • 12:49 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property --rebuild-all-terms (T172776)
  • 12:32 elukey: restart pdfrender on scb1002
  • 12:20 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T172776)
  • 12:15 gehel: restarting wdqs-updater on wdqs1001
  • 12:14 elukey: stop eventlogging on eventlog1001 to test kafka consumer failures
  • 11:22 Amir1: start of ladsgroup@terbium:~$ timeout 3500s /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item >>/tmp/rebuildTermSqlIndex.log 2>&1 (T171460)
  • 10:12 elukey: update librdkafka1* on notebook100[12] and stat1003
  • 09:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 as the main recentchanges/watchlist s6 role (duration: 00m 47s)
  • 08:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 at 50% load (duration: 00m 46s)
  • 08:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 with limited load (duration: 00m 46s)
  • 07:53 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1098 (duration: 00m 47s)
  • 07:41 elukey: stop puppet on cp3032 (cache::text) to set varnishkafka-webrequest logging to debug
  • 07:34 Amir1: stopped the script and re-running without --deduplicate-terms (T171460)
  • 07:21 Amir1: start of ladsgroup@terbium:~$ time mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --entity-type=property --deduplicate-terms (T171460)
  • 06:12 elukey: alert users with big home directories for stat1005 disk alarms (will erase data later on only if they don't answer)
  • 06:12 elukey: restart pdfrender on scb1003
  • 03:55 mutante: phab1001 /usr/local/bin/community_metrics.sh | /usr/local/bin/project_changes.sh creating stats mails to admins (which failed before) (T163938)
  • 03:25 mutante: phab1001 /srv/phab/tools/public_task_dump.py to create dump, was failed cron due to missing /srv/dumps/
  • 02:53 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 8 02:53:13 UTC 2017 (duration 6m 53s)
  • 02:46 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 06m 59s)
  • 02:36 ejegg: enabled donation queue consumer
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 25s)
  • 00:46 ejegg: disabled donation queue consumer

2017-08-07

  • 21:28 urandom: T172384: Upgrading Cassandra to 3.11.0-wmf1 in dev environment (build patched to disable in-built heap dumping)
  • 19:51 ejegg: updated CiviCRM from f24ba78 to 200abc2
  • 19:05 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212 - Turn on CirrusSearch MLR AB test (duration: 00m 46s)
  • 19:00 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 2) (duration: 00m 46s)
  • 18:59 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 1) (duration: 00m 46s)
  • 18:53 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: T171212 - Update CirrusSearch AB test rescore profiles (duration: 00m 46s)
  • 18:44 ebernhardson@tin: Synchronized wmf-config/Wikibase-labs.php: T112606 - beta only - Add copyright info for Wikidata API (duration: 00m 46s)
  • 18:43 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172630 - Enable wgMinervaEnableSiteNotice for kowiki (duration: 00m 46s)
  • 18:38 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)
  • 18:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CirrusSearch/: T169498 limit phrase token count, T172464 constant boost ltr queries (duration: 00m 58s)
  • 18:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172594 - Translate sitename for nl.wikinews (duration: 00m 47s)
  • 18:05 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Grant autopatrol to editor in en.wikibooks - T172561 (duration: 00m 47s)
  • 17:57 mutante: phab2001 - re-enabling puppet, but closing firewall for 80/443
  • 17:56 jynus: stopping slave and reparitioning db1098
  • 17:10 marostegui: Restart s7 instance on db1102 to pick up new replication filters - T172693
  • 17:05 gehel@tin: Finished deploy [wdqs/wdqs@da33919]: (no justification provided) (duration: 02m 28s)
  • 17:02 gehel@tin: Started deploy [wdqs/wdqs@da33919]: (no justification provided)
  • 16:44 marostegui: Restart s7 instance on db1069 to pick up new replication filters - T172693
  • 16:37 XioNoX: manually restarted varnish on cp1099
  • 15:10 thcipriani: restarting jenkins for plugin upgrade
  • 14:51 gehel: reducing elasticsearch eqiad concurrent rebalance to 4 (from 8)
  • 14:38 elukey: updated librdkafka1 and ++1 to 0.9.4.1 on hafnium
  • 14:32 mutante: phab2001 - stopping Apache,schedule downtime for http and puppet
  • 14:22 herron: mx[1,2]001, fermium: Installed libmail-dkim-perl and restarted spamassassin service - T172689
  • 13:15 jynus: reboot db1098
  • 12:39 _joe_: restarting pdfrender on scb1001, T159922
  • 12:39 elukey: restart kafka on kafka1018 to force it out of the kafka topic leaders - T172681
  • 12:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2074 - T171321 (duration: 00m 45s)
  • 12:08 gehel: deploying https://gerrit.wikimedia.org/r/#/c/299825/ - some logs will be lost during logstash restart
  • 10:02 marostegui: Add dbstore2002:3313 to tendril - T171321
  • 09:47 jynus: stopping db1050's mysql and cloning it to db1089
  • 09:06 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 (was 120) on all the analytics kafka brokers - T136094
  • 09:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 after fixing: linter, page and watchlist tables (duration: 00m 47s)
  • 08:12 marostegui: Force BBU re-learn on db1016 - T166344
  • 07:02 marostegui: Stop replication on db2065 to reimport: page, linter and watchlist tables
  • 07:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 to reimport: page, linter and watchlist tables (duration: 00m 47s)
  • 06:38 marostegui: Stop MySQL on db2074 - T171321
  • 06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2074 - T171321 (duration: 00m 46s)
  • 06:33 marostegui: Stop replication on db2075 - T170662
  • 06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2073 - T171321 (duration: 00m 47s)
  • 06:20 marostegui: Force BBU re-learn on db1016 - T166344
  • 02:57 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 7 02:57:42 UTC 2017 (duration 6m 42s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 56s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 10m 16s)

2017-08-06

  • 13:17 elukey: powercycle mw2256 - com2 frozen - T163346
  • 13:13 elukey: restart pdfrender on scb1002
  • 06:18 ebernhardson@tin: Synchronized wmf-config/PoolCounterSettings.php: T169498: Reduce cirrus search pool counter to 200 parallel requests cluster wide (duration: 02m 54s)
  • 01:28 chasemp: conf2002:~# service etcdmirror-conftool-eqiad-wmnet restart (not sure what else to do the service failed)

2017-08-05

  • 14:40 Reedy: created oauth tables on foundationwiki T172591
  • 14:13 reedy@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaMaintenance/createExtensionTables.php: add oauth (duration: 00m 48s)

2017-08-04

  • 23:51 mutante: phab2001 - removed outdated /etc/hosts entries, that fixed rsync, syncing /srv/repos/ from phab1001
  • 23:35 mutante: phab2001 rebooting
  • 23:35 mutante: phab2001 - installing various package upgrades, apt-get autoremove old kernel images
  • 23:12 mutante: "reserved" UID 498 for phd on https://wikitech.wikimedia.org/wiki/UID | phab2001: find -exec chown to fix all the files , restart cron
  • 23:04 mutante: phab2001 - changing UID/GID for phd user from 997:997 to 498:498 to make it match phab1001, to fix rsync breaking permissions. (rsync forces --numeric-ids when fetching from and rsyncd configured with chroot=yes). chown -R phw:www-data /srv/repos/
  • 22:37 ejegg: restarted donations and refund queue consumers
  • 21:44 ejegg: stopped donations and refund queue consumers
  • 21:24 urandom: T172384: Disabling Puppet in dev environment to prevent unattended Cassandra restarts
  • 20:19 mutante: renewing SSL cert for status.wm.org (just like wikitech-static, but that one didnt have monitoring?)
  • 20:02 mutante: wikitech-static-ord - apt-get install certbot
  • 19:41 ejegg: updated CiviCRM from f1fd7f0 to f24ba78
  • 19:38 mutante: renaming graphite varnish director/fixing config, running puppet on cache misc, tested on cp1045
  • 18:17 andrewbogott: switched most cloud instance to new puppetmasters, as per https://phabricator.wikimedia.org/T171786
  • 11:46 marostegui: Deploy schema change directly on s3 master for maiwikimedia - T172485
  • 11:30 marostegui: Deploy schema change directly on s3 master for kbpwiki - T172485
  • 11:14 marostegui: Deploy schema change directly on s3 master for dinwiki - T172485
  • 10:14 marostegui: Deploy schema change directly on s3 master for atjwiki - T172485
  • 10:05 marostegui: Stop replication on db2073 for maintenance
  • 09:22 marostegui: Add dbstore2002 to tendril - T171321
  • 09:19 marostegui: Deploy schema change directly on s3 master for techconductwiki - T172485
  • 08:35 marostegui: Deploy schema change directly on s3 master for hiwikiversity - T172485
  • 08:19 marostegui: Deploy schema change directly on s3 master for wikimania2018wiki - T172485
  • 08:04 marostegui: Sanitize wikimania2018wiki on sanitarium and sanitarium2 - T155041
  • 07:47 marostegui: Stop MySQL on db2073 to copy its data to dbstore2002 - https://phabricator.wikimedia.org/T171321
  • 07:47 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T171321 (duration: 00m 47s)
  • 07:07 moritzm: installing imagemagick regression security updates on trusty
  • 06:47 marostegui: Sanitize hiwikiversity on sanitarium and sanitarium2 - T171829
  • 05:23 mutante: phab1001 sudo ip addr del 10.64.32.186/32 dev eth0 (T172478)
  • 02:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet
  • 02:28 dzahn@neodymium: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
  • 02:15 mutante: phab1001 can't talk to mx servers via IPv6, but works via IPv4. iridium and other mailservers can also talk IPv6 to it. why? it did not change even when stopping ferm on client and on server it allows from anywhere. workaround for now was to hardcode IPv4 IP in phab config. (T163938)
  • 02:13 twentyafterfour: outgoing phab mail is working again
  • 01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chown -R phd:www-data /srv/repos
  • 01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chmod -x /usr/local/sbin/sync-srv-repos
  • 01:12 mutante: phab1001 - stopped and started exim, which is now running with same options as iridium
  • 00:53 twentyafterfour: starting phd to shut up icinga
  • 00:40 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn CirrusSearch MLR test back off (duration: 00m 47s)
  • 00:35 twentyafterfour: phabricator web service is also up
  • 00:34 mutante: phab1001 - service IPs switched - puppet ran - ssh-phab service up
  • 00:30 twentyafterfour: phab1001 chown -R phd:phd /srv/repos
  • 00:28 twentyafterfour: testing phd on phab1001
  • 00:24 mutante: phab1001 sudo ip addr add 2620:0:861:103:10:64:32:186/128 dev eth0
  • 00:24 mutante: phab1001 sudo ip addr add 10.64.32.186/32 dev eth0
  • 00:23 mutante: iridium sudo ip addr del 2620:0:861:ed1a::3:16/128 dev lo
  • 00:23 mutante: iridiium sudo