Renaming of Cloud Services products is in progress.
You may see mixed references to Tool Labs -> Toolforge, Labs -> Cloud VPS. Edits are welcome.

Server Admin Log

From Wikitech
Jump to: navigation, search

2017-10-17

  • 23:31 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/Citoid/modules/ve.ui.CiteFromIdInspector.css: SWAT: ve.ui.CiteFromIdInspector: Fix CSS for context menus after changes in OOjs UI T178324 (duration: 00m 50s)
  • 23:29 thcipriani@tin: Synchronized php-1.31.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/styles/widgets/ve.ui.MWMediaInfoFieldWidget.css: SWAT: ve.ui.MWMediaInfoFieldWidget: Fix positioning of icons T178415 (duration: 00m 50s)
  • 23:22 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/shell/Command.php: SWAT: Shell\Command: Better walltime fallback T178314 (duration: 00m 51s)
  • 23:15 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Remove setting no longer in MediaWiki (duration: 00m 50s)
  • 23:06 mutante: disabling Icinga notifications for service recommendation_api on scb hosts - please remember to re-enable once ticket is resolved (T178445) (https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=recommendation_api%20endpoints)
  • 22:28 ejegg: updated CiviCRM from fb513cc to 7c26791
  • 21:55 foks: Removed 2FA from User:Amakuru
  • 21:39 robh: cp4026 being returned to service post memory replacement. puppet runs fine and all icinga checks are green. repooled.
  • 21:23 robh: cp4026 coming down for memory swap, no point in maint moding the system since other cp checks will alert anyhow
  • 20:14 mutante: gerrit2001 - re-enabled puppet, attempting restart with --slave after gerrit:384758
  • 19:54 mutante: cobalt - re-enabled puppet, ran puppet, systemd says gerrit is active(running)
  • 19:51 mutante: cobalt - stopping gerrit service, confirming pid file gets removed, move /etc/init.d/gerrit to /root
  • 19:50 mutante: short gerrit downtime for systemd change coming up
  • 19:23 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4
  • 19:06 mutante: gerrit2001 - rebooting to verify systemd change and service comes back properly
  • 18:53 mutante: gerrit2001: stopping the gerrit service
  • 18:52 mutante: cobalt: temp disable puppet | gerrit2001: stop puppet
  • 18:22 paravoid: rebooting sodium for kernel/distro upgrade
  • 18:14 paravoid: dist-upgrading sodium to stretch
  • 18:12 demon@tin: Finished scap: bootstrap wmf.4 (duration: 48m 41s)
  • 17:23 demon@tin: Started scap: bootstrap wmf.4
  • 16:49 demon@tin: Pruned MediaWiki: 1.31.0-wmf.2 [keeping static files] (duration: 01m 24s)
  • 16:35 gehel: restart tilerator / tileratorui on all maps cluster after deployment of latest fix - T177389
  • 16:34 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389 (duration: 01m 42s)
  • 16:32 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: fix tileshell not exiting - T177389
  • 15:46 _joe_: killed a stuck gerrit process on cobalt
  • 14:47 bd808: wikitech: Cleaned up 'importers' user_group entries (T171682)
  • 14:25 Amir1: end of cleaning up ores_classification table in plwiki, start of ruwiki (T159753)
  • 14:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 - T174509 T177772 (duration: 00m 45s)
  • 14:05 Amir1: start of cleaning up ores_classification table in plwiki (T159753)
  • 14:00 zeljkof: EU SWAT finished
  • 13:59 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Fix typo in UserTesting config var (take 2) (T177502) (duration: 00m 45s)
  • 13:47 zeljkof: EU SWAT continues
  • 13:31 zeljkof: stopping EU SWAT
  • 13:29 jynus: killing commonswiki script on tin
  • 13:17 zfilipin@tin: Synchronized php-1.31.0-wmf.3/extensions/UniversalLanguageSelector/maintenance/ULSCompactLinksDisablePref.php: SWAT: Remove the 20 edits threshold from ULSCompactLinksDisablePref.php (duration: 00m 46s)
  • 12:53 volans: upgrading cumin masters to cumin 1.2.2-1
  • 12:40 moritzm: upgrading mwdebug* to hhvm-wikidiff 1.5.1
  • 11:47 marostegui: Optimize recentchanges on db1102 for s4 - T174509
  • 11:47 marostegui: Optimize templatelinks and pagelinks on db1102 for s4 - T174509
  • 11:45 Amir1: end of cleaning up ores_classification table in hewiki, start of nlwiki (T159753)
  • 11:39 Amir1: end of cleaning up ores_classification table in fiwiki, start of hewiki (T159753)
  • 11:32 jynus: test-installing proxysql on wasat T175672
  • 11:24 Amir1: end of cleaning up ores_classification table in fawiki, start of fiwiki (T159753)
  • 11:12 gehel: restarting wdqs-updater on wdqs1004 - T175919
  • 10:40 Amir1: end of cleaning up ores_classification table in etwiki, start of fawiki (T159753)
  • 10:36 Amir1: end of cleaning up ores_classification table in cswiki, start of etwiki (T159753)
  • 10:26 Amir1: start of cleaning up ores_classification in cswiki (T159753)
  • 10:16 akosiaris: removing akosiaris's yubikey from network devices
  • 09:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 - T174509 T177772 (duration: 00m 46s)
  • 09:23 marostegui: Stop MySQL on db2084 for multi-instance testing
  • 09:16 mobrovac@tin: Finished deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3 (duration: 07m 34s)
  • 09:08 mobrovac@tin: Started deploy [restbase/deploy@7f228d4]: Remove /page/revision and add history metrics end points, part #3
  • 09:08 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2 (duration: 25m 57s)
  • 08:45 gehel: restarting blazegraph on wdqs1004 for GC tuning (adding -XX:+G1PrintRegionLivenessInfo) - T175919
  • 08:42 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points, part #2
  • 08:40 mobrovac@tin: Finished deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805 (duration: 01m 05s)
  • 08:39 mobrovac@tin: Started deploy [restbase/deploy@3bcd1b7]: Remove /page/revision and add history metrics end points - T158100 T175805
  • 07:54 marostegui: Stop MySQL and reboot db2081 to see if it works fine - T178140
  • 07:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2086 to the config - T170662 (duration: 00m 45s)
  • 07:42 marostegui: Stop MySQL on db2079 to clone db2086 - T170662
  • 07:36 moritzm: rebooting boron for kernel update
  • 07:31 ema: upgrade misc_eqiad to varnish 5 T177233
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 - T174509 (duration: 00m 45s)
  • 07:11 marostegui: Optimize templatelinks and pagelinks on db1067 - T174509
  • 07:10 marostegui: Optimize enwiki.ores_classification on db1067 - T159753
  • 06:55 moritzm: installing libffi security updates on trusty (Debian already fixed)
  • 06:07 marostegui: Stop replication in sync on db1072 and db1103 for data drift fixing - T164488
  • 05:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1034 - T174509 (duration: 00m 45s)
  • 05:54 marostegui: Optimize pagelinks and templatelinks on db1034 - T174509
  • 05:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 - T174509 T177772 (duration: 00m 45s)
  • 05:46 marostegui: Optimize pagelinks, templatelinks and recentchanges on db1085 T177772 T174509
  • 05:42 marostegui: Optimize pagelinks, templatelinks and ores_classification on db1095 - T174509 T159753
  • 05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T174509 (duration: 00m 45s)
  • 05:38 marostegui: Optimize recentchanges on db1081 - T177772
  • 05:37 marostegui: Optimize pagelinks and templatelinks on db1081 - T174509
  • 05:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 45s)
  • 05:32 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
  • 05:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1073 - T174509 (duration: 00m 46s)
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 17 02:32:30 UTC 2017 (duration 6m 57s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 08m 23s)
  • 00:34 ejegg: updated CiviCRM from 824c2d8 to fb513cc

2017-10-16

  • 23:49 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/CheckUser/specials/SpecialCheckUser.php: SWAT: Revert "Revert "Restore checkuser-userlinks when exist"" T170507 (duration: 00m 46s)
  • 23:46 ebernhardson: increase elasticsearch log levels for transport to debug to help track down T167410
  • 23:39 ejegg: updated civicrm from b953064 to 824c2d8
  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/Echo/includes/ContainmentSet.php: SWAT: ContainmentSet: Use strict comparison for array_search() T177825 (duration: 01m 45s)
  • 20:35 chasemp: disable puppet around cloud things for a refactor merge
  • 20:20 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 (duration: 02m 59s)
  • 19:59 demon@tin: Synchronized wmf-config/CommonSettings.php: Ief0ea01f (duration: 00m 46s)
  • 19:58 jynus: sudo apt install wmf-mariadb101-client on terbium for debugging an issue
  • 19:57 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Ief0ea01f (duration: 00m 46s)
  • 19:56 demon@tin: Synchronized tests/cirrusTest.php: Ief0ea01f (duration: 00m 47s)
  • 18:18 thcipriani@tin: Synchronized wmf-config/Wikibase.php: SWAT: Add configuration for statement indexing for Wikidata T175199 (duration: 00m 47s)
  • 17:35 ariel@tin: Finished deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files (duration: 00m 03s)
  • 17:35 ariel@tin: Started deploy [dumps/dumps@2b9326b]: do the minimum for latestlinks job, no updating other files
  • 17:08 gehel@tin: Finished deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates (duration: 02m 20s)
  • 17:06 gehel@tin: Started deploy [wdqs/wdqs@b42eb0d]: WDQS GUI uipdates
  • 16:56 ottomata: deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223
  • 16:39 joal@tin: Finished deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix (duration: 04m 42s)
  • 16:35 joal@tin: Started deploy [analytics/aqs/deploy@339674c]: Deploy new mediawiki-history endpoints (alpha version) after monitorin fix
  • 16:23 joal@tin: Finished deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix (duration: 06m 56s)
  • 16:16 joal@tin: Started deploy [analytics/aqs/deploy@aa7ef0d]: Deploy new mediawiki-history endpoints (alpha version) after node-modules fix
  • 16:09 ema: upgrade misc_codfw to varnish 5 T177233
  • 16:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T174509 T177772 (duration: 00m 46s)
  • 15:51 gehel: killing stuck tileshell process on maps-test2001 - T177389
  • 15:51 gehel@tin: Finished deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389 (duration: 00m 20s)
  • 15:50 gehel@tin: Started deploy [tilerator/deploy@f3b26f3]: testing tileshell fix - T177389
  • 15:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T174509 (duration: 00m 46s)
  • 15:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T174509 T177772 (duration: 00m 47s)
  • 14:34 twentyafterfour: deploying hotfix for T178068 and T178107
  • 14:30 moritzm: uploaded wikidiff 1.5.1 to apt.wikimedia.org for jessie/experimental
  • 14:08 hashar: European swat is complete
  • 14:08 hashar@tin: Synchronized wmf-config/abusefilter.php: (T177760) Enable AbuseFilter profiler | (T177761) Allow AbuseFilter to block & grant relevant permissions to permit administrators to manage the restricted option. (duration: 00m 47s)
  • 13:50 moritzm: upgrading deployment-mediawiki0[56] to wikidiff2 1.5.1
  • 13:45 hashar@tin: Synchronized wmf-config/: [cirrus] Prepare profiles for the recall A/B test on enwiki - T177502 (duration: 00m 48s)
  • 13:44 moritzm: upgrading deployment-mediawiki04 to wikidiff2 1.5.1
  • 13:42 hashar@tin: Synchronized wmf-config: [cirrus] Disable A/B test of MLR on testing on 18 wikis with > 1% of search traffic - T177490 (duration: 00m 48s)
  • 13:38 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add additional namespaces to search results for bnwikisource - T178041 (duration: 00m 49s)
  • 12:59 gehel: restart cassandra on maps1001
  • 12:38 ema: upgrading eqiad LVSs to pybal 1.14.2 (T178149, T177815)
  • 11:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T174509 (duration: 00m 46s)
  • 11:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1070 - T174509 (duration: 00m 46s)
  • 11:10 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 10:48 Amir1: cleaning up ores_classification table in wikidatawiki (T159753)
  • 10:03 ema: upgrading codfw LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:54 ema: upgrading esams LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:44 ema: upgrading ulsfo LVSs to pybal 1.14.2 (T178149, T177815)
  • 09:43 ema: pybal 1.14.2 uploaded to apt.w.o
  • 09:40 moritzm: restarting cassandra on restbase1007 to pick up NSS security update
  • 09:39 moritzm: restarting cassandra on cerium/xenon/praseodymium to pick up NSS security update
  • 09:37 gehel: restarting elasticsearch on elastic1017
  • 09:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T174509 T177772 (duration: 00m 46s)
  • 09:07 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1088 - https://phabricator.wikimedia.org/T174509 https://phabricator.wikimedia.org/T177772
  • 09:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 - T174509 T177772 (duration: 00m 46s)
  • 08:53 marostegui: Stop MySQL on db1050 as it will be decommissioned - T178162
  • 08:46 marostegui: Remove db1050 from tendril - T178162
  • 08:31 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)
  • 07:56 marostegui: Stop db1103 and db1038 at the same position to fix data drifts - T164488
  • 07:42 moritzm: installing NSS security updates
  • 07:30 marostegui: Stop db1103 and db1072 at the same position to fix data drifts - T164488
  • 07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 - T164488 (duration: 00m 46s)
  • 07:19 marostegui: Optimize table ores_classification on db1073 - T159753
  • 07:14 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2084 to the config - T170662 (duration: 00m 46s)
  • 07:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2084 to the config - T170662 (duration: 00m 47s)
  • 06:25 marostegui: Stop MySQL on db2079 to clone db2084 from it - T170662
  • 06:12 marostegui: Optimize pagelinks and templatelinks on db1094 - T174509
  • 06:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 - T174509 T177772 (duration: 00m 46s)
  • 06:04 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1093 - T174509 T177772
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1070 - T174509 (duration: 00m 46s)
  • 05:59 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1070 - T174509
  • 05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T174509 T177772 (duration: 00m 46s)
  • 05:53 marostegui: Optimize recentchanges, pagelinks and templatelinks on db1084 - T174509 T177772
  • 05:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T174509 (duration: 00m 46s)
  • 05:47 marostegui: Optimize pagelinks and templatelinks on db1074 - T174509
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1073 - T174509 (duration: 00m 47s)
  • 05:37 marostegui: Optimize templatelinks and pagelinks on db1073 - T174509
  • 05:36 marostegui: Optimize templatelinks and pagelinks on db1073 - 174509
  • 02:35 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 16 02:35:34 UTC 2017 (duration 6m 36s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 07m 44s)

2017-10-15

  • 11:15 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916

2017-10-14

  • 03:52 paravoid: manually cleaning up pdns_metric_cron from all dnsrecursors (hydrogen/chromium/acamar/achernar/maerlant/nescio)

2017-10-13

  • 23:04 mutante: scb1001 - systemctl restart pdfrender (Icinga said: pdfrender Connection refused)
  • 20:36 mutante: gerrit2001 - removing hhvm package (which is now possible without also removing scap, thanks thcipriani), apt-get autoremove to remove more unused packages (T178039)
  • 18:52 joal@tin: Finished deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version (duration: 15m 02s)
  • 18:40 joal@tin: (no justification provided)
  • 18:37 joal@tin: Started deploy [analytics/aqs/deploy@0375c34]: Deploy new mediawiki-history endpoints-alpha version
  • 16:54 otto@tin: Finished deploy [analytics/refinery@28db253]: (no justification provided) (duration: 05m 50s)
  • 16:48 otto@tin: Started deploy [analytics/refinery@28db253]: (no justification provided)
  • 15:31 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 (duration: 00m 46s)
  • 14:59 mutante: added cparle to wmf LDAP group
  • 14:38 jynus: stopping slave on db1056 and optimize recentchanges
  • 14:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1100, depool db1056, fix db1098 weight (duration: 00m 46s)
  • 14:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 (duration: 00m 46s)
  • 13:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 (duration: 00m 47s)
  • 12:57 godog: upload scap 3.7.1-1 - T127762
  • 11:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 11:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 10:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 00m 47s)
  • 10:22 godog: roll-restart pybal to apply https://gerrit.wikimedia.org/r/#/c/383997/ - T178149
  • 10:04 jynus: stopping slave on db1098 and optimize recentchanges
  • 09:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098 (duration: 00m 46s)
  • 09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2082 to the config - T170662 (duration: 00m 46s)
  • 09:22 Amir1: a small clean up of ores_classification table in wikidatawiki (T159753)
  • 09:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2082 to the config - T170662 (duration: 00m 47s)
  • 09:04 godog: restart pybal on lvs1003 to test logstash depool - T178078
  • 08:57 marostegui: Stop db1103 and db1038 in sync for more checksumming - T164488
  • 08:34 ema: upgrade pybal on lvs1003 to 1.14.1 T177815
  • 08:18 ema: upgrade pybal on lvs1006 to 1.14.1 T177815
  • 08:15 jynus: restarting commons wiki recentchanges purge of wb entries T177772
  • 08:11 ema@neodymium: conftool action : set/pooled=yes; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
  • 08:11 ema@neodymium: conftool action : set/pooled=no; selector: name=logstash1001.eqiad.wmnet,service=logstash-gelf,dc=eqiad
  • 08:10 ema: depool/repool logstash1001 (service=logstash-gelf) T178078
  • 07:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2081 to the config - T170662 (duration: 00m 46s)
  • 07:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2081 to the config - T170662 (duration: 00m 47s)
  • 06:23 marostegui: Stop MySQL on db2080 to copy its data to db2081 - T170662
  • 00:22 mutante: authdns servers: deleted /etc/ganglia/conf.d/gndsd.pyconf & /usr/lib/ganglia/python_modules_gdnsd.py - removing ganglia stats

2017-10-12

  • 23:58 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Restore thanks limit on pl.wikipedia (T169268) (duration: 00m 47s)
  • 23:50 cwd: re-enabled omnimail jobs
  • 23:35 eileen1: update from 2d6668b to b953064
  • 23:25 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Add logging for email blocks (T175419) (duration: 00m 46s)
  • 23:13 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch test wikis to HTML5 fragment mode in links (T152540) (duration: 00m 47s)
  • 22:44 demon@tin: Synchronized php-1.31.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php: silence notices (duration: 00m 47s)
  • 21:36 cwd: disabled omnimail jobs
  • 19:47 ejegg: updated CiviCRM from 47de976 to 2d6668b
  • 19:46 chasemp: disable puppet for lab*services* for merge of 383896
  • 19:18 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.3
  • 18:28 mutante: added Pablo (pgrass) to LDAP group 'wmde' (T177599)
  • 18:15 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/ORES/includes/Hooks.php: Make watchlist queries suck less (duration: 00m 50s)
  • 18:01 demon@tin: Synchronized php-1.31.0-wmf.3/extensions/JsonConfig/includes/JCCache.php: Silence junk warnings (duration: 00m 50s)
  • 15:17 mobrovac@tin: Started restart [electron-render/deploy@8dd5f13]: Electron hanging - T174916
  • 15:11 mutante: gerrit2001 - rebooting for kernel upgrade
  • 14:57 mutante: netmon1002 - arming keyholder for rancid deploy
  • 14:52 mutante: netmon1002 (librenms, rancid) - rebooting for kernel upgrade
  • 14:47 moritzm: rebooting kubernetes workers to new 4.9.51 kernel
  • 14:41 moritzm: rebooting kubernetes staging servers to new 4.9.51 kernel
  • 14:29 moritzm: rebooting kubernetes masters to new 4.9.51 kernel
  • 13:54 moritzm: updating image scalers to librsvg 2.40.18
  • 13:48 elukey: deployed the new Analytics Public Druid cluster - T176223
  • 13:45 moritzm: updating remaining thumbor servers to librsvg 2.40.18
  • 13:40 ladsgroup@tin: Synchronized wmf-config/Wikibase.php: SWAT: Remove deprecated config variable for Wikibase (T129475) (duration: 01m 02s)
  • 13:33 moritzm: updating thumbor1001 to librsvg 2.40.18
  • 13:13 moritzm: rolling restart of logstash to pick up thumbor logs (T150734)
  • 13:07 zeljkof: EU SWAT finished
  • 12:56 moritzm: uploaded librsvg 2.40.18 for jessie-wikimedia to apt.wikimedia.org
  • 11:46 ema: upgrading cp3010 to Varnish 5 T177233
  • 11:27 Amir1: finished cleanup of ores_classification in wikidatawiki, still needs more work (T159753)
  • 10:03 ema: lvs1005: upgrade pybal to 1.14.1 T177815
  • 10:02 ema: pybal 1.14.1 uploaded to apt.w.o
  • 09:40 ema: upgrading cp3008 to Varnish 5 T177233
  • 09:06 moritzm: installing nettle update from stretch 9.2 point release
  • 09:01 moritzm: installing ncurses security updates
  • 08:25 moritzm: installing gnutls28 update from stretch 9.2 point release
  • 08:02 Amir1: starting cleanup of ores_classification in wikidatawiki (T159753)
  • 07:51 gehel: reinitialize cassandra on maps-test2004 to test vector tiles - T153282
  • 07:40 moritzm: installing at-spi2-core update from stretch 9.2 point release
  • 07:24 moritzm: installing gnupg2 update from stretch 9.2 point release
  • 07:18 legoktm: purging html5-misnesting linter entries from database on terbium (T178040)
  • 03:25 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 12 03:25:17 UTC 2017 (duration 7m 17s)
  • 03:18 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.3) (duration: 15m 57s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 39s)
  • 00:30 twentyafterfour: phabricator update finished (a while ago, I forgot to log it)
  • 00:08 twentyafterfour: upgrading phabricator to #phab-2017-10-11
  • 00:05 mutante: netmon2001 arm keyholder for rancid deploy
  • 00:01 twentyafterfour: Preparing to take phabricator offline for a hopefully momentary upgrade.

2017-10-11

  • 23:55 mutante: netmon2001 - smokeping back up - the reboot, as a side-effect, also resolved the systemd-icinga-alert that was pending in Icinga for a while with comments
  • 23:52 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART II (duration: 00m 50s)
  • 23:50 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Enable Vector print logo and print styles on all wikis T169732 PART I (duration: 00m 50s)
  • 23:47 mutante: netmon2001 (smokeping) - kernel upgrade, schedule downtime, reboot
  • 23:40 thcipriani@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 49s)
  • 23:38 mutante: netmon1003 - servermon back up - apt-get autoremove to drop unrequired python packages
  • 23:37 thcipriani@tin: Synchronized php-1.31.0-wmf.3/skins/Vector/ResourceLoaderLessModule.php: SWAT: Print logo should use an absolute URI T177800 (duration: 00m 50s)
  • 23:35 mutante: netmon1003 - cant seem to reboot normally, attempting via gnt-instance command
  • 23:31 mutante: netmon1003 (servermon) - upgrading kernel (jessie), schedule icinga downtime, rebooting, short downtime for servermon itself
  • 23:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on nowiki T177989 (duration: 00m 50s)
  • 23:27 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Parser: Switch from Tidy to Remex on fawiki T176150 (duration: 00m 50s)
  • 23:23 thcipriani@tin: Synchronized php-1.31.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/styles/ve.init.MWVESwitchConfirmDialog.css: SWAT: Fix WikiEditor mode switcher widget (duration: 00m 50s)
  • 23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 50s)
  • 23:18 mutante: releases1001 upgrading kernel, installed linux-image-amd64, took out of cache::misc, upgrade libnss3, scheduled downtime, rebooting ...
  • 23:17 thcipriani@tin: Synchronized php-1.31.0-wmf.3/includes/specials/SpecialUnblock.php: SWAT: Fix unblocking autoblocks T177952 (duration: 00m 51s)
  • 22:01 volans: uploaded cumin_1.2.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 21:44 robh: cp4009 shutdown accidental, will boot back up immediately
  • 21:13 mutante: releases2001 - scheduled downtime, rebooting
  • 21:07 mutante: running puppet on cache misc hosts to temp remove codfw backend of releases.wm.org, then rebooting releases2001 for kernel upgrade
  • 20:22 subbu: updated Parsoid to ddf7b293 (T138492, T135667, T177612)
  • 20:18 ssastry@tin: Finished deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293 (duration: 08m 42s)
  • 20:17 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 02m 54s)
  • 20:14 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:13 hashar@tin: Finished deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044 (duration: 00m 02s)
  • 20:13 hashar@tin: Started deploy [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:11 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:10 bsitzmann@tin: Finished deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301) (duration: 05m 25s)
  • 20:09 ssastry@tin: Started deploy [parsoid/deploy@938e305]: Updating Parsoid to ddf7b293
  • 20:07 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:05 bsitzmann@tin: Started deploy [mobileapps/deploy@15a52ba]: Update mobileapps to c045afc (T177301)
  • 20:04 hashar@tin: Started restart [jobrunner/jobrunner@a20d043]: Services now exit(0) when catching a signal - T168044
  • 20:01 chasemp: disable puppet around cloud things to roll refactor out slowly
  • 19:29 mutante: releases1001 - same as 2001, upgraded jenkins to 2.73.2, kept existing config (T177962)
  • 19:27 mutante: releases2001 - upgraded jenkins to 2.73.2, kept existing config (vs overwriting with package config)
  • 19:20 mutante: apt: reprepro copy stretch-wikimedia jessie-wikimedia jenkins
  • 19:16 mutante: releases1001/releases2001 - apt-get autoremove to drop non-required python packages
  • 19:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.3
  • 19:02 demon@tin: Synchronized php: symlink swap (duration: 00m 49s)
  • 18:47 gehel: killing stuck osm replication on maps-test2001
  • 18:23 demon@tin: Synchronized wmf-config/Wikibase-labs.php: no-op, labs only (duration: 00m 59s)
  • 18:19 demon@tin: Finished scap: wmf.3 bootstrap (duration: 44m 56s)
  • 17:58 herron: ran /home/faidon/update-netboot-stretch.sh on puppetmaster1001
  • 17:49 moritzm: uploaded jenkins 2.73.2 for jessie-wikimedia to apt.wikimedia.org
  • 17:34 demon@tin: Started scap: wmf.3 bootstrap
  • 17:32 demon@tin: Synchronized php: symlink swap to wmf.2, somehow got missed (duration: 00m 45s)
  • 17:29 demon@tin: Synchronized scap/plugins/prep.py: No-op (duration: 01m 49s)
  • 17:09 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 (duration: 02m 20s)
  • 17:06 demon@tin: Pruned MediaWiki: 1.30.0-wmf.16 (duration: 02m 40s)
  • 16:54 demon@tin: Pruned MediaWiki: 1.31.0-wmf.1 [keeping static files] (duration: 01m 44s)
  • 16:24 hasharAway: Upgrade jenkins Maven integration plugin to 3.0 - T177962
  • 16:15 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/380942 - T150734
  • 16:12 hasharAway: Upgrading Jenkins CI T177962
  • 16:09 gehel: decommission maps-test2004 from its cassandra cluster (free a node to test vector tiles) - T153282
  • 16:09 XioNoX: resuming PDU upgrade
  • 15:39 elukey: drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002
  • 15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T174509 (duration: 00m 47s)
  • 15:21 _joe_: stopped redis on ocg* T177931
  • 15:08 _joe_: ocg*: stop ocg; mv /srv/deployment /srv/stale, update-rc.d ocg disable, rm /etc/init/ocg.conf - T177931
  • 14:55 _joe_: manually removing IPVS entries for ocg on eqiad LBs, T177931
  • 14:47 _joe_: rolling restart of low-traffic pybals in eqiad for T177931
  • 14:36 moritzm: installing dbus update from stretch 9.2 point release
  • 13:40 hashar: European SWAT completed
  • 13:39 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Remove compact language links dblist for simplicity (duration: 00m 46s)
  • 13:38 hashar@tin: Synchronized docroot: Remove compact language links dblist for simplicity (duration: 00m 48s)
  • 13:37 hashar@tin: Synchronized dblists: Remove compact language links dblist for simplicity (duration: 00m 47s)
  • 13:36 hashar@tin: Synchronized wmf-config/CommonSettings.php: Remove compact language links dblist for simplicity (duration: 00m 47s)
  • 13:29 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Configure wmgBabelMainCategory for the Dinka Wikipedia (duration: 00m 47s)
  • 13:27 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable mapframe on eswiki - T177695 (duration: 00m 47s)
  • 13:24 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Enable constraint checks on qualifiers and references - T176863 (duration: 00m 47s)
  • 13:21 moritzm: installing db5.3 security updates
  • 13:03 godog: test graphite-web patch on labmon1001 - T177747
  • 12:54 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Disable Statement usage tracking on cawiki (T151717) (duration: 00m 47s)
  • 12:53 marostegui: Start recentchanges purge on s4 primary master T177772
  • 12:48 marostegui: Kill recentchanges purge on s4 primary master - https://phabricator.wikimedia.org/T177772
  • 12:44 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (temp) Disable Statement usage tracking on cawiki (T151717) (duration: 00m 48s)
  • 12:38 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Statement usage tracking on cawiki and cewiki (T151717) (duration: 00m 47s)
  • 12:32 hoo@tin: Synchronized wmf-config/: Move WB client "disabledUsageAspects" setting into $wmgWikibaseDisabledUsageAspects (duration: 00m 48s)
  • 12:30 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Introduce $wmgWikibaseDisabledUsageAspects (duration: 00m 47s)
  • 12:27 hoo@tin: scap aborted: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects (duration: 02m 34s)
  • 12:25 hoo@tin: Started scap: wmf-config/InitialiseSettings.php Introduce $wmgWikibaseDisabledUsageAspects
  • 11:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 - T174509 (duration: 00m 50s)
  • 11:49 ema: upgrading cp3007 to Varnish 5 T177233
  • 11:34 moritzm: installing ruby1.9 security updates on trusty hosts
  • 11:19 akosiaris: re-enable OTRS stats group T176221
  • 11:11 jynus: starting purge of ruwiki.recentchanges T177772
  • 10:55 akosiaris: OTRS upgrade done T176221
  • 10:44 akosiaris: starting OTRS upgrade T176221
  • 10:24 jynus: stopping dbstore2001:s3 for maintenance T177908
  • 10:19 moritzm: installing curl security updates
  • 09:32 moritzm: installing libxfont security updates
  • 09:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T174509 (duration: 00m 46s)
  • 09:27 marostegui: Optimize pagelinks and templatelinks on db1087 - T174509
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T174509 (duration: 00m 47s)
  • 09:04 moritzm: installing curl security updates on app server canaries (along with HHVM restart)
  • 08:50 moritzm: installing ffmpeg security updates
  • 08:38 elukey: reboot kafka-jumbo hosts for kernel updates
  • 08:37 marostegui: Stop replication in sync on db1103 and db1038 to checksum their data - T164488
  • 08:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103 - T164488 (duration: 00m 47s)
  • 07:46 jynus: restart commonswiki recentchanges purging T177772
  • 07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1072 into the vslow and dump service for s3 - T172679 (duration: 00m 46s)
  • 07:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T174509 (duration: 00m 47s)
  • 07:02 marostegui: Optimize templatelinks and pagelinks on db1086 - T174509
  • 06:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T174509 (duration: 00m 47s)
  • 06:57 marostegui: Optimize templatelinks and pagelinks on db1082 - T174509
  • 06:14 marostegui: Set db1068 s4 primary master to read-only OFF - T168661
  • 06:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 to writable mode after maintenance - T168661 (duration: 00m 47s)
  • 06:07 marostegui: Deploy alter table on s4 primary master - T168661
  • 06:01 marostegui: Set read-only on db1068 s4 primary master - T168661
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Set s4 in read-only for maintenance - T168661 (duration: 00m 47s)
  • 05:48 marostegui: Package update on db1068 s4 primary master - T168661
  • 05:35 marostegui: Disable puppet on db1068 (s4 primary master) - T168661
  • 05:35 marostegui: Disable puppet on db1068 (s4 primary master)
  • 05:13 marostegui: Disable alerts on s4 for 2 hours - T168661
  • 05:06 marostegui: Dump buffer pool on s4 primary master db1068 - T168661
  • 05:03 kartik@tin: Finished deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515 (duration: 02m 57s)
  • 05:00 kartik@tin: Started deploy [cxserver/deploy@a79b38c]: Update cxserver to 273b515
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 11 02:32:46 UTC 2017 (duration 6m 39s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 52s)

2017-10-10

  • 23:53 thcipriani@tin: Synchronized wmf-config: SWAT: Disable OCG services PART II T177795 (duration: 00m 48s)
  • 23:52 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable OCG services PART I T177795 (duration: 00m 47s)
  • 23:41 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Collection/RenderingAPI.php: SWAT: Do not request render if renderer not configured T177795 (duration: 00m 47s)
  • 23:39 XioNoX: upgrading ps1-a2-eqiad - T175341
  • 23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: revert Enable Vector print logo on test wiki (duration: 00m 47s)
  • 23:21 thcipriani@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:12 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: wikitech: Align "contentadmin" and "sysop" permissions T171208 (duration: 00m 48s)
  • 23:08 eileen1: ran the group clean up script (reduced smart groups)
  • 22:28 eileen1: changed civicrm smart group cache timeout to 5 civicrm/admin/setting/search?reset=1
  • 21:15 bblack: esams repooling - T167840
  • 19:30 krinkle@tin: Synchronized docroot/noc/: Clean up noc base.css (duration: 00m 48s)
  • 19:07 krinkle@tin: Synchronized docroot/noc/: Clean up noc/index CSS (duration: 00m 47s)
  • 19:00 XioNoX: starting the work for T167840
  • 18:41 mutante: logstash*: delete /usr/lib/ganglia/python_modules/elasticsearch_monitoring.py and /etc/ganglia/conf.d/elasticsearch.pyconf via cumin (T177225)
  • 17:32 XioNoX: depooling esams from DNS for T167840
  • 17:26 gehel: shutting down and restarting elasticsearch on relforge1001 for testing - T170378
  • 17:25 krinkle@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery.ui/: I717f2580e3aae (duration: 00m 47s)
  • 16:19 milimetric@tin: Finished deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder (duration: 16m 01s)
  • 16:03 milimetric@tin: Started deploy [analytics/refinery@f4a0a33]: Deploying mostly for the new script in the bin folder
  • 15:48 jynus: starting purge of commonswiki.recentchanges T177772
  • 15:39 thcipriani@tin: Synchronized README: noop deployment to test new logstash-checker.py host (duration: 00m 47s)
  • 15:27 jynus: upgrading neodymium and sarin to mariadb-client 10.1.28
  • 15:23 ema: cp1008 upgraded to varnish 5 T168529
  • 15:00 godog: test stretch 9.2 upgrade on ms-fe2005 - T177739
  • 14:59 cmjohnson1: lvs1007 swapping NIC card
  • 14:53 godog: test stretch 9.2 upgrade on ms-be2013 - T177739
  • 14:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T174509 (duration: 00m 47s)
  • 14:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T174509 (duration: 00m 47s)
  • 14:31 marostegui: Optimize table ores_classifications on db1080 - T159753
  • 14:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight - T172679 (duration: 00m 47s)
  • 14:24 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 00m 47s)
  • 14:22 elukey: add druid public cluster's IPs to analytics-in4 on cr1/cr2 - T177511
  • 13:56 aude@tin: Synchronized wmf-config/throttle.php: Add throttle rule T177835 (duration: 02m 16s)
  • 13:37 aude@tin: Synchronized wmf-config/Wikibase-production.php: Remove test wiki echo configs from Wikibase config (duration: 01m 31s)
  • 13:20 aude@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on gawiki (duration: 01m 50s)
  • 13:11 gehel: restarting rsyslog on mw1180 - T177833
  • 11:32 moritzm: downgrading HHVM on deployment-mediawiki04 to HHVM 3.18.2 temporarily (to further narrow down a problem with the new wikidiff package)
  • 11:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T174509 (duration: 05m 02s)
  • 11:31 dcausse: upgrading relforge to elastic 5.5.2
  • 11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1071 - T174509 (duration: 01m 21s)
  • 10:33 akosiaris: T177511 switch druid100[456] to private1-x-eqiad VLANs
  • 10:26 Amir1: start of cleaning up ores_classification in wikidatawiki (T159753)
  • 08:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2080 to the config - T170662 (duration: 00m 46s)
  • 08:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2080 to the config - T170662 (duration: 00m 47s)
  • 07:04 marostegui: Stop MySQL on db2079 to clone db2080 - https://phabricator.wikimedia.org/T170662
  • 06:35 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s3 - T153033
  • 06:27 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s1 - T153033
  • 06:22 marostegui: Stop MySQL on db1038 to transfer its data to db1072 - T172679
  • 06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1038 - T172679 (duration: 00m 47s)
  • 06:05 marostegui: Stop MySQL on db1072 to move it to s3 - T172679
  • 05:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T174509 (duration: 00m 47s)
  • 05:53 marostegui: Optimize templatelinks and pagelinks on db1079 - T174509
  • 05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1071 - T174509 (duration: 00m 48s)
  • 05:45 marostegui: Optimize templatelinks and pagelinks on db1071 - T174509
  • 05:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174509 (duration: 00m 47s)
  • 05:38 marostegui: Optimize templatelinks and pagelinks on db1076 - T174509
  • 05:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T174509 (duration: 00m 48s)
  • 05:32 marostegui: Optimize templatelinks and pagelinks on db1080 - T174509
  • 02:35 eileen2: update civicrm from 00da4b0 to 47de976
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 10 02:32:57 UTC 2017 (duration 6m 41s)
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 08m 49s)
  • 01:21 eileen2: update civicrm from 276cbfd to 00da4b0

2017-10-09

  • 22:18 hoo: Removed empty left-over xmldatadumps/public/other/wikibase/wikidatawiki/20171007 (https://dumps.wikimedia.org/wikidatawiki/entities/20171007/)
  • 19:53 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
  • 19:51 hashar: morning swat completed
  • 19:50 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo/maintenance/updatePerUserBlacklist.php: Sync a live hack in Echo updatePerUserBlacklist https://gerrit.wikimedia.org/r/#/c/383182/ - T173475 (duration: 00m 47s)
  • 19:48 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable new print styles on Vector in test wiki - T169732 (duration: 00m 47s)
  • 19:22 hashar@tin: Synchronized wmf-config/CommonSettings.php: REVERT: Enable new print styles on Vector - T169732 (duration: 00m 49s)
  • 19:19 hashar@tin: scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 19:17 hashar@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: Reapply "Use User Ids instead of User Names for Echo Mute"" - T173475 (duration: 00m 52s)
  • 18:57 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on hi.wikiversity - T177690 (duration: 00m 47s)
  • 18:52 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable NewUserMessage for SUL accounts too on dty.wiki - T177688 (duration: 00m 47s)
  • 18:45 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled user group to sd.wikipedia - T177141 (duration: 00m 47s)
  • 18:00 ottomata: stopping druid services on druid1004 (not 1006)
  • 17:59 ottomata: stopping druid services on druid1006
  • 17:28 ottomata: stopping druid services on druid1005
  • 16:34 ottomata: stopping druid services on druid1006
  • 16:34 ottomata: beginning decom and reinstall process for druid1004-1006 -- T176223
  • 15:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T174509 (duration: 00m 47s)
  • 13:53 zeljkof: EU SWAT finished
  • 13:53 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/: SWAT: Revert "[tech-debt] Remove usage of FuzzyLikeThis in favor of simple fuzzy match" (T177727) (duration: 00m 57s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disallow most file types from upload to enwikivoyage (T176647) (duration: 00m 47s)
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix amwikimedia site name (T176042) (duration: 00m 47s)
  • 13:10 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177737) (duration: 00m 47s)
  • 12:47 ema: restart pybal on eqiad load balancers to pick up bgp-med config change T165584
  • 12:38 ema: restart pybal on codfw load balancers to pick up bgp-med config change T165584
  • 12:31 ema: restart pybal on ulsfo load balancers to pick up bgp-med config change T165584
  • 12:26 ema: restart pybal on esams load balancers to pick up bgp-med config change T165584
  • 11:59 joal@tin: Finished deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy (duration: 10m 24s)
  • 11:49 joal@tin: Started deploy [analytics/refinery@c3812c2]: Analytics regular weekly deploy
  • 11:42 hashar: Upgrading Jenkins - T168644
  • 11:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 46s)
  • 11:22 moritzm: installing apt updates from stretch 9.2 point release
  • 11:22 marostegui: Optimize ores_classification table on db1083 - T159753
  • 11:19 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Actually disable injecting RC everywhere but commonswiki and ruwiki (duration: 00m 46s)
  • 11:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 - T174509 (duration: 00m 47s)
  • 11:08 moritzm: installing whois updates from stretch 9.2 point release
  • 11:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090 - T174509 (duration: 00m 46s)
  • 10:54 moritzm: installing expect updates from stretch 9.2 point release
  • 10:50 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable injecing RC records for commonswiki and ruwiki (T171027) (duration: 00m 47s)
  • 10:38 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/Wikidata/extensions/Wikibase/client: Re-instate $wgWBClientSettings[injectRecentChanges] (T171027) (duration: 00m 56s)
  • 10:35 hoo: Started dumpwikidatajson.sh on snapshot1007 after T169680 disruptions
  • 10:30 volans: rebooting snapshot1007 for NFS stuck - T169680
  • 10:21 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s7 - T153033
  • 10:21 volans: rebooting snapshot1006 for NFS stuck - T169680
  • 10:15 elukey: rebooting snapshot1001 for NFS stuck - T169680
  • 10:00 volans: rebooting snapshot1005 for NFS stuck - T169680
  • 09:51 volans: rebooting dataset1001 for NFS stuck - T169680
  • 09:51 ema: test setting bgp med on lvs3001/3003 T165584
  • 09:50 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s5 - T153033
  • 09:43 hashar: restarting Jenkins. Deadlock in SSHSlave plugin that causes memory to leak quite rapidly - T177749
  • 09:34 moritzm: installing vim security updates
  • 09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 - T170662 (duration: 00m 47s)
  • 08:56 volans: disabled puppet on 'stat100[5-6]*,snapshot100[1,5-7]*,dataset1001*' to manually recover from nfsd stuck - T169680
  • 08:52 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s4 - T153033
  • 08:51 marostegui: Optimize pagelinks and templatelinks on db1092 - T174509
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174509 (duration: 00m 47s)
  • 08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096 - T174509 (duration: 00m 47s)
  • 08:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1066 on s1 to and move db1072 to s3 instead - T172679 (duration: 00m 47s)
  • 08:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Move db1066 from s1 to s3 - T172679 (duration: 01m 25s)
  • 07:33 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s2 - T153033
  • 07:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2079 to the config - T170662 (duration: 00m 47s)
  • 07:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2079 to the config - T170662 (duration: 00m 47s)
  • 07:01 ema: cp4022 - backend restart, mailbox lag, cache_upload
  • 06:56 marostegui: Stop MySQL on db2075 to use it to clone db2079 - T170662
  • 06:56 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 - T170662 (duration: 00m 46s)
  • 06:25 marostegui: Drop moodbar_feedback and moodbar_feedback_response from s6 - T153033
  • 06:14 marostegui: Optimize pagelinks and templatelinks on db1039 - T174509
  • 06:11 marostegui: Optimize pagelinks and templatelinks on db1050 - T174509
  • 06:08 marostegui: Optimize pagelinks and templatelinks on db1096 - T174509
  • 06:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096 - T174509 (duration: 00m 47s)
  • 05:59 marostegui: Optimize pagelinks and templatelinks on db1091 - T174509
  • 05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T174509 (duration: 00m 47s)
  • 05:53 marostegui: Optimize pagelinks and templatelinks on db1090 - T174509
  • 05:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090 - T174509 (duration: 00m 47s)
  • 05:44 marostegui: Optimize pagelinks and templatelinks on db1083 - T174509
  • 05:38 marostegui: Stop MySQL on db1083 to upgrade it
  • 05:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 47s)
  • 02:48 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 9 02:48:21 UTC 2017 (duration 6m 54s)
  • 02:41 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 13m 49s)

2017-10-08

  • 21:40 hoo: Killed "dumpwikidatardf.sh truthy nt" (wikidata truthy dump) on snapshot1007, got stuck after T169680.
  • 11:28 volans: ack-ed puppet not running on stat100[5-6],snapshot100[1,5-7] due to NFSD stuck on dataset1001 - T169680
  • 08:19 elukey: restart varnish backend on cp4026 to stop 503s

2017-10-07

  • 22:46 reedy@tin: Synchronized php-1.31.0-wmf.2/resources/: T177705 (duration: 00m 48s)
  • 22:44 reedy@tin: Synchronized php-1.31.0-wmf.2/includes/specials/SpecialBlock.php: T177705 (duration: 00m 49s)

2017-10-06

  • 18:58 ejegg: updated CiviCRM from 197b22d to 276cbfd
  • 18:38 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/: Unbreak preference changes (T177524) (duration: 00m 47s)
  • 18:23 arlolra@tin: Finished deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf (duration: 12m 41s)
  • 18:10 arlolra@tin: Started deploy [parsoid/deploy@e437c93]: Updating Parsoid to 772e11bf
  • 15:30 moritzm: repooling mw2145 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
  • 15:25 moritzm: repooling mw2203 (API server). it was previously depooled, but there's no tickets or SAL entries indicating hardware maintenance, so it was probably simply forgotten to repool it
  • 12:54 moritzm: repooling mw2248-mw2250 (job runners). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
  • 11:46 bblack: cp1* (eqiad caches) - upgrade to nginx-1.13.5-1+wmf1~jessie1 to match all the other sites
  • 11:34 moritzm: repooling mw2244/mw2245 (image scalers). they were previosly depooled, but there's no tickets or SAL entries indicating hardware maintenance, so they were probably simply forgotten to repool
  • 11:21 moritzm: installing gdk-pixbuf security updates
  • 11:10 elukey: rolling restart of all the druid daemons on druid100[1-6] to pick up new logging changes
  • 11:03 moritzm: installing git security updates on trusty (Debian already fixed)
  • 10:51 moritzm: upgrading HHVM on script runners to HHVM 3.18.5
  • 10:47 moritzm: upgrade remaining video scalers in codfw to HHVM 3.18.5
  • 10:43 elukey: restart replication (mysql+ eventlogging_sync) on dbstore1002 after mysql restart
  • 10:17 moritzm: upgrade remaining job runners in codfw to HHVM 3.18.5
  • 09:53 elukey: stop replication (eventlogging_sync + mysql) on dbstore1002 as prep step for mysql restart
  • 09:17 hashar: Restarted the CI Jenkins for plugin upgrades
  • 09:01 moritzm: upgrade remaining app servers in codfw to HHVM 3.18.5
  • 08:40 moritzm: upgrade remaining image scalers in codfw to HHVM 3.18.5
  • 08:09 ema: upgrade pybal to 1.14.0 on eqiad primary LVSs
  • 07:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1076 original weight - T174054 (duration: 00m 48s)
  • 07:23 ema: upgrade pybal to 1.14.0 on eqiad secondary LVSs
  • 07:18 moritzm: upgrade remaining API servers in codfw to HHVM 3.18.5
  • 07:11 legoktm: live hacking on mwdebug1002
  • 07:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
  • 06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1076 weight - T174054 (duration: 00m 47s)
  • 06:35 ema: set max_connections=0 on cp3032 varnish-be
  • 06:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1106 to the config - T172679 (duration: 00m 47s)
  • 06:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1106 to the config - T172679 (duration: 00m 47s)
  • 06:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight - T174054 (duration: 00m 46s)
  • 05:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 - T174509 (duration: 00m 48s)
  • 05:35 marostegui: Stop MySQL on db1076 for an upgrade

2017-10-05

  • 23:40 ejegg: updated CiviCRM from cca1a9d to 197b22d
  • 23:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/resources/lib/jquery/jquery.migrate.js: sync https://gerrit.wikimedia.org/r/#/c/382530 and https://gerrit.wikimedia.org/r/#/c/382529 (duration: 00m 47s)
  • 23:34 XioNoX: upgrading ps1-d2-eqiad.mgmt.eqiad.wmnet (unused)
  • 23:23 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/BaseInterwikiResolver.php: sync https://gerrit.wikimedia.org/r/#/c/382625/ (duration: 00m 47s)
  • 23:21 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT https://gerrit.wikimedia.org/r/#/c/382611/ (duration: 00m 47s)
  • 22:47 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/includes/: Sync CirrusSearch extension again for good measure (duration: 00m 51s)
  • 22:43 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: Sync CirrusSearch extension again for good measure (duration: 00m 57s)
  • 22:39 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: sync https://gerrit.wikimedia.org/r/#/c/382474/ (duration: 00m 47s)
  • 22:37 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch/: deploy https://gerrit.wikimedia.org/r/#/c/382618/ (duration: 00m 58s)
  • 22:13 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 19s)
  • 22:13 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
  • 22:04 bd808@tin: Finished deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681) (duration: 00m 33s)
  • 22:04 bd808@tin: Started deploy [striker/deploy@fbb9019]: Prevent tools from being named with invalid Kubernetes namespace labels (T176681)
  • 21:35 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
  • 21:32 twentyafterfour@tin: Synchronized php-1.31.0-wmf.2/extensions/CirrusSearch: sync CirrusSearch to deploy https://gerrit.wikimedia.org/r/#/c/382605/ refs T177535 (duration: 01m 06s)
  • 21:22 ejegg: updated CiviCRM from 59e3e00 to cca1a9d
  • 21:13 twentyafterfour: Getting the train back on track: moving to wmf.2 as soon as https://gerrit.wikimedia.org/r/#/c/382605/1 merges
  • 20:47 twentyafterfour: 1.31.0-wmf.2 is now blocked by T177535
  • 20:28 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.1 refs T174358
  • 20:26 twentyafterfour: rolling back to 1.31.0-wmf.1
  • 20:11 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.31.0-wmf.2 refs T174358
  • 20:10 twentyafterfour: Promote all wikis to 1.31.0-wmf.2 refs T174358 (currently there are no blockers and no significant logspam)
  • 19:59 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 55s)
  • 19:58 andrewbogott: upgrading wikitech-static to REL1_30
  • 19:55 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: revert "Use User Ids instead of User Names for Echo Mute" (duration: 00m 52s)
  • 19:49 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/VisualEditor/extension.json: SWAT: Move 'parentheses' message to MWSaveDialog's modules T177446 (duration: 00m 50s)
  • 19:45 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict: SWAT: Dont register simulate page when not enabled as a BF (duration: 00m 52s)
  • 19:05 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 55s)
  • 18:58 thcipriani@tin: Synchronized php-1.31.0-wmf.2/extensions/Echo: SWAT: Use User Ids instead of User Names for Echo Mute T173475 (duration: 00m 56s)
  • 18:53 bblack: cp3* upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 18:38 urandom: restbase-ng: lowering compactionthroughput to 4 (current/5 jbod devices)
  • 18:26 thcipriani@tin: Synchronized wmf-config: SWAT: Enable structured change filters by default on all wikis except FlaggedRevs wikis using non-protection modes T177445 (duration: 00m 54s)
  • 18:13 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on all wikis T124742 (duration: 00m 52s)
  • 17:27 cmjohnson1: disable puppet db1024
  • 16:53 moritzm: upgrade remaining app servers in eqiad to HHVM 3.18.5
  • 16:16 XioNoX: failure test for pfw3-codfw
  • 15:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T174054 (duration: 00m 50s)
  • 15:25 marostegui: Pulling out one disk from db1078 - T174054
  • 15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T174054 (duration: 00m 51s)
  • 15:16 marostegui: Pulling out one disk from db1076 - T174054
  • 14:53 hashar: All Jenkins infra is now on Java 8 - T162828
  • 14:52 marostegui: Sanitize db1095 for amwikimedia - T176043
  • 14:48 hashar: Restarting Jenkins
  • 14:27 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
  • 13:57 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable Statement usage tracking on kowiki and trwiki (T151717) (duration: 00m 50s)
  • 13:51 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: (no justification provided) (duration: 00m 50s)
  • 13:49 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Document task reference for amwikimedia logo (T176042) (duration: 00m 54s)
  • 13:43 Amir1: ladsgroup@terbium:~$ mwscript createAndPromote.php --wiki=amwikimedia --createAndPromote=sysop,bureaucrat 'Ladsgroup' "password removed obviously" --force (T176042)
  • 13:35 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Fix amwikimedia logo (T176042) (duration: 00m 50s)
  • 13:26 ladsgroup@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaMessages/i18n/wikimediaprojectnames: (no justification provided) (duration: 00m 49s)
  • 13:15 godog: test upgrade node-exporter in thumbor/swift/services/puppet3-diffs/monitoring projects - T166561
  • 13:11 ladsgroup@tin: Synchronized wmf-config/interwiki.php: Create amwikimedia (T176042) (duration: 00m 51s)
  • 13:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 50s)
  • 12:58 ladsgroup@tin: Synchronized multiversion/MWMultiVersion.php: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:53 ladsgroup@tin: Synchronized static/images/project-logos/: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:52 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:50 ladsgroup@tin: rebuilt wikiversions.php and synchronized wikiversions files: Create amwikimedia (T176042)
  • 12:49 ladsgroup@tin: Synchronized dblists: Create amwikimedia (T176042) (duration: 00m 50s)
  • 12:36 ladsgroup@tin: Synchronized dblists: (no justification provided) (duration: 00m 51s)
  • 12:22 moritzm: installing poppler security updates on trusty
  • 11:57 moritzm: upgrade remaining eqiad video scalers to HHVM 3.18.5
  • 11:29 jynus: rebuilding globalimagelinks table on dbstore1001:s4
  • 10:35 hashar: contint2001: switched java to version 8 as well as all the jre/jdk utilities managed by alternatives - T162828
  • 10:26 moritzm: upgrade remaining eqiad API servers to HHVM 3.18.5
  • 10:23 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: REVERT: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 50s)
  • 10:22 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: REVERT: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
  • 10:17 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/extension.json: Dont register simulate page when not enabled as a BF PT2/3 (duration: 00m 50s)
  • 10:16 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/TwoColConflictHooks.php: Dont register simulate page when not enabled as a BF PT1/3 (duration: 00m 51s)
  • 10:14 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/TwoColConflict/includes/SpecialConflictTestPage/SpecialConflictTestPage.php: Fix SimulateTwoColEditConflict when using a page with ns (duration: 00m 50s)
  • 10:09 addshore@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.2 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 50s)
  • 10:07 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: 31-wmf.1 WikimediaEvents: WMDE: update $campaignStartTimestamp (duration: 00m 52s)
  • 10:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T174054 (duration: 00m 50s)
  • 09:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099 - T174509 (duration: 00m 50s)
  • 08:54 marostegui: Stop MySQL on db1104 to clone db1106 - T172679
  • 08:53 gehel: rolling restart of wdqs to pick up new GC options - T175919
  • 08:41 moritzm: upgrade remaining eqiad job runners to HHVM 3.18.5
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1105 to the config - T172679 (duration: 00m 50s)
  • 08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1105 to the config - T172679 (duration: 00m 52s)
  • 08:29 marostegui: Optimize pagelinks and templatelinks on s5 - db1105 - T174509
  • 08:21 moritzm: upgrade remaining eqiad image scalers to HHVM 3.18.5
  • 08:12 marostegui: Optimize templatelinks and pagelinks on s1 s2 s4 s5 s6 and s7 on labsdb1009 - T174509
  • 08:03 jynus: putting labsdb1009 under maintenance
  • 07:54 moritzm: upgrade mw1221-mw1235 (API servers) to HHVM 3.18.5
  • 07:26 legoktm: running batched query of "DELETE FROM linter WHERE linter_cat=12" on all wikis to clear out mostly bogus html5-misnesting category
  • 07:11 ema: update pybal to 1.14.0 on esams primary LVSs
  • 07:06 ema: update pybal to 1.14.0 on esams secondary LVSs
  • 07:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099 - T174509 (duration: 00m 52s)
  • 07:01 marostegui: Optimize templatelinks and pagelinks on db1099 - T174509
  • 06:56 moritzm: upgrade mw1238-mw1258 (app servers) to HHVM 3.18.5
  • 06:40 marostegui: Stop MySQL on db1104 to clone db1105 from it - T172679
  • 06:32 marostegui: Optimize templatelinks and pagelinks on db1069 - T174509
  • 06:31 marostegui: Optimize templatelinks and pagelinks on db1053 - T174509
  • 06:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T174509 (duration: 00m 50s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 49s)
  • 06:24 marostegui: Optimize templatelinks and pagelinks on db1089 - T174509
  • 06:17 marostegui: Optimize s2.templatelinks and enwiki.pagelinks on codfw master db2017 (without replication) - T174509
  • 06:13 marostegui: Optimize enwiki.templatelinks and enwiki.pagelinks on codfw master db2048 (without replication) - T174509
  • 06:01 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2035 and db2056 - T174509 (duration: 00m 52s)
  • 05:57 elukey: restart varnish backend on cp3040
  • 05:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097 - T174509 (duration: 01m 09s)
  • 05:42 elukey: restart varnish backend on cp3030
  • 02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Oct 5 02:59:18 UTC 2017 (duration 7m 23s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 09m 01s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 09m 51s)
  • 00:06 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: T176652 (duration: 00m 49s)
  • 00:04 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T176652 (duration: 00m 50s)

2017-10-04

  • 23:55 eileen: update process_control to 10161f8
  • 23:53 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/resources/jquery.embedPlayer.js: Fix jqmigrate warning (T169385) (duration: 00m 50s)
  • 23:51 catrope@tin: Synchronized php-1.31.0-wmf.2/extensions/ORES/includes/Hooks.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:49 catrope@tin: Synchronized php-1.31.0-wmf.2/resources/src/: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:36 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilter.php: ORES highlights for RCFilters (T172757) (duration: 00m 50s)
  • 23:32 catrope@tin: Synchronized php-1.31.0-wmf.2/includes/changes/ChangesListFilterGroup.php: Fix PHP fatal when transcluding Special:Recentchanges (T176236) (duration: 00m 50s)
  • 23:32 Krinkle: Upgrading xhgui on tungsten (operations/software/xhgui.git) from 0.5.2 to 0.8.1 (matching version used in operations/mediawiki-config vendor) - new feature: flamegraphs
  • 23:29 catrope@tin: Synchronized wmf-config/throttle.php: Remove expired rules (duration: 00m 50s)
  • 23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile on ptwiki (T177336) (duration: 00m 51s)
  • 23:19 catrope@tin: Synchronized static/images/mobile/copyright/wikipedia-wordmark-en.svg: Remove unnecessary id attributes (T175670) (duration: 00m 50s)
  • 23:18 catrope@tin: Synchronized php-1.31.0-wmf.2/skins/MinervaNeue/: Correct feature phone threshold detection (T176286) (duration: 00m 53s)
  • 22:35 ejegg: changed DonationInterface cURL timeout for all processors to 12 sec
  • 21:09 arlolra: Updated Parsoid to 477942c3 (T177115)
  • 21:07 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 21:03 arlolra@tin: Finished deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3 (duration: 10m 01s)
  • 20:53 arlolra@tin: Started deploy [parsoid/deploy@617cdb3]: Updating Parsoid to 477942c3
  • 20:33 arlolra@tin: Finished deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3 (duration: 00m 58s)
  • 20:32 arlolra@tin: Started deploy [parsoid/deploy@5b80254]: Updating Parsoid to 477942c3
  • 20:18 pnorman: Start regenerating map tiles on codfw for z13-z14 - T176252
  • 19:55 twentyafterfour@tin: Synchronized php: group1 wikis to 1.31.0-wmf.2 refs T174358 (duration: 00m 48s)
  • 19:55 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.2 refs T174358
  • 19:38 pnorman: Start regenerating map tiles on eqiad for z13-z14 - T176252
  • 19:25 twentyafterfour: Deploying MediaWiki 1.31.0-wmf.2 to Group 1 wikis refs T174358
  • 19:06 bblack: cp2* (all codfw caches): upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 18:39 ottomata: reverting roll out of LVS service druid-analytics.svc.eqiad.wmnet - this wont' work with hosts inside of Analytics VLAN
  • 18:23 niharika29@tin: Synchronized php-1.31.0-wmf.2/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 50s)
  • 18:22 niharika29@tin: Synchronized php-1.31.0-wmf.1/skins/Vector/: Do not special-case ULS and Not logged in in RTL in personal bar T48947, T177312 (duration: 00m 51s)
  • 18:09 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on most group1 wikis (T124742) (duration: 00m 51s)
  • 17:37 elukey: enabled basic ACLs on the Kafka Jumbo cluster - T173493
  • 17:32 bblack: text@ulsfo - upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 17:32 ottomata: deploying new LVS service for druid-analytics-broker
  • 17:16 moritzm: uploaded icu52 to stretch-wikimedia (co-installable forward port of jessie's ICU for stretch to be used for HHVM)
  • 17:06 bblack: cp4022-26 (now all of upload@ulsfo) - upgrade nginx to 1.13.5-1+wmf1~jessie1
  • 16:58 krinkle@tin: Synchronized php-1.31.0-wmf.2/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayer.loader.js: T175506 (duration: 00m 52s)
  • 16:29 bblack: lvs1001-3: base package upgrades, autoremove, +python-requests
  • 16:24 bblack: recdns: upgrade base packages
  • 16:23 volans: deleted 'R:File = /usr/local/bin/wmf-reimage' on matching hosts T176955
  • 16:20 bblack: maelant: upgrade base packages
  • 16:11 bblack: lvs1004-6: base package upgrades, autoremove, +python-requests
  • 16:06 jynus: performing schema change on labsdb1009/10/11
  • 16:01 ejegg: disabled CiviCRM dedupe jobs
  • 15:40 bblack: acamar (recdns@codfw) - base package upgrades
  • 15:39 bblack: lvs1007-12: base package upgrades, autoremove, +python-requests
  • 15:37 bblack: hydrogen (recdns@eqiad) - base package upgrades
  • 15:35 marostegui: Power off db2010 to decommission it - T175685
  • 15:35 bblack: recdns: apt autoremove -> install python-requests
  • 15:32 bblack: caches: apt autoremove -> install python-requests
  • 15:28 marostegui: Disable puppet on db2010 - it will be decommissioned - T175685
  • 15:24 andrewbogott: rebuilding labvirt1016
  • 14:57 ema: cp4025: restart varnish backend
  • 14:47 XioNoX: replacing faulty VC cable on fasw-c-eqiad
  • 14:45 bblack: base package upgrades (+autoremove) on lvs3001-2
  • 14:43 bblack: base package upgrades (+autoremove) on lvs3004
  • 14:41 bblack: base package upgrades (+autoremove) on lvs3003
  • 14:24 bblack: upgrading nginx to 1.13.5-1+wmf1~jessie1 on cp4021 (first live cache, upload@ulsfo)
  • 14:18 bblack: base package upgrades (+autoremove) on lvs2*
  • 14:10 bblack: base package upgrades (+autoremove) on lvs4*
  • 13:47 zeljkof: EU SWAT finished
  • 13:46 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
  • 13:45 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 50s)
  • 13:41 zfilipin@tin: Synchronized php-1.31.0-wmf.2/extensions/Translate/TranslateHooks.php: php-1.31.0-wmf.2/extensions/Translate/tag/PageTranslationHooks.php SWAT: Revert "Use Language type hint in hooks" (T177352) (duration: 00m 52s)
  • 13:26 zeljkof: one more commit for EU SWAT
  • 13:26 marostegui: Optimize table ores_classification on enwiki codfw master db2048 with replication - might generate lag - T159753
  • 13:22 marostegui: Optimize enwiki.ores_classification on db2069
  • 13:06 zeljkof: EU SWAT finished
  • 13:05 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: New throttle rule (T177370) (duration: 00m 53s)
  • 12:59 jynus: upgrade and reboot labsdb1009
  • 12:56 marostegui: Optimize templatelinks and pagelinks tables on s2 and s7 - labsdb1010 - T174509
  • 12:38 moritzm: upgrading app servers in codfw to HHVM 3.18.5
  • 12:21 moritzm: upgrading image scalers mw1293-mw1295 to HHVM 3.18.5
  • 12:12 moritzm: upgrading HHVM on deployment servers to 3.18.5
  • 12:10 elukey: added two new mediawiki videoscalers - mw1307/1318
  • 12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1318.eqiad.wmnet
  • 12:09 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
  • 12:00 gehel: mediawiki now uses the LVS endpoint for logstash - T175242
  • 11:49 moritzm: upgrading job runners mw1161-mw1167 to HHVM 3.18.5
  • 11:34 moritzm: installing ghostscript security updates
  • 11:13 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1087 original weight (duration: 00m 47s)
  • 10:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1087 weight (duration: 00m 51s)
  • 10:29 jynus: upgrade and reboot labsdb1011
  • 10:22 jynus: draining labsdb1011 connections
  • 10:18 akosiaris: T177374, fully depool wtp1001-wtp1024
  • 10:18 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
  • 10:03 moritzm: install libidn security updates
  • 10:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 with low weight (duration: 00m 50s)
  • 09:54 _joe_: restarting hhvm on canaries (both appservers and canaries) T107128
  • 09:47 moritzm: upgrading API app servers mw1189-mw1208 to HHVM 3.18.5
  • 09:45 elukey: rolling restart of aqs nodes to pick up the new logstash lvs config
  • 09:39 hoo: Killed remaining dumpRdf runners on snapshot1007 after two crashed due to the db1087 depool. Auto-restart will do the rest.
  • 09:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1104 to the config - T172679 (duration: 00m 50s)
  • 09:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1104 to the config - T172679 (duration: 00m 51s)
  • 09:25 moritzm: installing ocaml security updates on trusty
  • 09:16 marostegui: Reboot db1087 to pick up new kernel
  • 09:09 _joe_: restarting hhvm on mwdebug*, T107128
  • 09:07 marostegui: Optimize templatelinks and pagelinks tables on db1097 (s4) - T174509
  • 09:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T174509 (duration: 00m 50s)
  • 08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp10([01][0-9]|2[0-4]).eqiad.wmnet
  • 08:40 mobrovac@tin: Finished deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3 (duration: 10m 05s)
  • 08:36 moritzm: upgrading app servers mw1180-mw1188, mw1209-mw1220 to HHVM 3.18.5
  • 08:30 mobrovac@tin: Started deploy [restbase/deploy@8eb758a]: Switch the mobile feeds to the new storage schema and Cassandra 3
  • 08:28 marostegui: Stop MySQL on db2010 as it will be decommissioned - T175685
  • 07:30 jynus: stopping s5 replication on dbstore1002 and converting wikidatawiki.wb_terms into TokuDB
  • 07:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
  • 07:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2010 from config as it will be decommissioned - T175685 (duration: 00m 48s)
  • 06:53 marostegui: Stop MySQL on db1087 to clone db1104 from it - T172679
  • 06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 - T172679 (duration: 00m 50s)
  • 06:30 marostegui: Optimize templatelinks and pagelinks tables on db1066 - T174509
  • 06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2035 and db2056 - T174509 (duration: 00m 50s)
  • 06:21 elukey: restart varnish backend on cp3043
  • 06:19 marostegui: Optimize pagelinks and templatelinks on dbstore2002 s1 - T174509
  • 06:16 elukey: restart varnish backend on cp3041
  • 06:13 marostegui: Optimize pagelinks and templatelinks tables on s7 codfw master (db2029) this will generate lag - T174509
  • 05:58 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2034, db2055 db2062 - T174509 (duration: 00m 51s)
  • 03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Oct 4 03:09:34 UTC 2017 (duration 7m 15s)
  • 03:02 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.2) (duration: 15m 20s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 44s)
  • 00:33 ejegg: re-enabled fundraising queue consumers
  • 00:29 ejegg: re-enabled CiviCRM de-dupe jobs
  • 00:24 ejegg: re-enabled civicrm cron and ingenico jobs
  • 00:20 ejegg: re-enabled fundraising stats and export jobs
  • 00:16 ejegg: re-enabled omnimail import jobs
  • 00:10 ejegg: re-enabled fundraising audit jobs
  • 00:07 ejegg: re-enabled pending queue consumer

2017-10-03

  • 23:52 ejegg: stopped all jobs for process-control upgrade
  • 23:46 pnorman: Start regenerating map tiles on eqiad for z0-z12 - T176252
  • 23:20 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175971 T176088 Enable RemexHTML on wikitech and eswikiversity (duration: 00m 51s)
  • 23:11 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: Pre-configure cirrussearch for 5.5.x upgrade (duration: 00m 51s)
  • 22:26 bsitzmann@tin: Finished deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519) (duration: 06m 14s)
  • 22:20 bsitzmann@tin: Started deploy [mobileapps/deploy@82aa7d6]: Update mobileapps to 5dc0c02 (T175762 T177001 T176525 T176517 T176519)
  • 20:29 pnorman: Start regenerating map tiles on codfw for z0-z12 - T176252
  • 20:17 demon@tin: Synchronized php-1.31.0-wmf.2/extensions/GlobalUserPage/includes/Hooks.php: fix broken thing (duration: 00m 51s)
  • 19:58 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.2
  • 19:33 demon@tin: Synchronized scap/plugins/prep.py: no-op (duration: 00m 51s)
  • 19:17 demon@tin: Finished scap: bootstrap wmf.2 (duration: 33m 50s)
  • 19:01 twentyafterfour: Starting MediaWiki train for 1.30.1-wmf.2
  • 18:43 demon@tin: Started scap: bootstrap wmf.2
  • 18:25 chasemp: change to private vlan for labweb1001 on asw-b-eqiad
  • 18:24 chasemp: change to private vlan for labweb1002 on asw2-d-eqiad
  • 18:24 chasemp: auth-dnsupdate on bahan for 382027
  • 16:51 ejegg: updated payments-wiki from c2a3d43 to b9859f6
  • 15:57 bblack: upgrading basic packages on role::cache::text
  • 15:50 bblack: upgrading basic packages on role::cache::upload
  • 15:42 bblack: upgrading basic packages on role::cache::misc
  • 15:29 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 06m 27s)
  • 15:24 _joe_: uploading blubber to jessie-wikimedia T175296
  • 15:23 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 15:04 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 05m 14s)
  • 14:59 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 14:56 mobrovac@tin: Finished deploy [restbase/deploy@02acc45]: (no justification provided) (duration: 04m 25s)
  • 14:56 bblack: upgrading basic packages on cp1065
  • 14:52 mobrovac@tin: Started deploy [restbase/deploy@02acc45]: (no justification provided)
  • 13:56 zeljkof: EU SWAT finished
  • 13:55 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/Translate/webservices/CxserverWebService.php: SWAT: Make CxserverWebService forward compatible (duration: 00m 45s)
  • 13:46 godog: upgrade grafana to 4.5.2 on krypton - T175980
  • 13:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust wgNamespacesToBeSearchedDefault for enwikibooks, fawikibooks and hewikisource (T176906 T176908 T176907) (duration: 00m 46s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Limit thanks for new users at pl.wikipedia to 3 per day" (T176174 T169268) (duration: 00m 46s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ShortUrl Extension on hiwikiversity (T177187) (duration: 00m 46s)
  • 13:25 marostegui: Optimize templatelinks and pagelinks tables on s1, s4 and s6 on labsdb1010 - T174509
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage Extension on hiwikiversity (T177188) (duration: 00m 46s)
  • 13:04 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T174764 (duration: 00m 47s)
  • 10:50 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: (no justification provided) (duration: 08m 38s)
  • 10:41 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: (no justification provided)
  • 08:10 Amir1: cleaning up ores_classification tables (T159753)
  • 08:05 marostegui: Deplo alter table on s3 codfw master (db2018) to add PK to the main tables - T163912
  • 08:02 mobrovac@tin: Finished deploy [restbase/deploy@65f519b]: Create new buckets for feed end points (duration: 09m 57s)
  • 07:52 mobrovac@tin: Started deploy [restbase/deploy@65f519b]: Create new buckets for feed end points
  • 07:20 gehel: killing stuck tileshell and cleaning up /srv/osm_expire on maps-test2001 - T175123
  • 07:05 elukey: restart varnish backend on cp3031 (503s)
  • 07:03 marostegui: Optimize pagelinks and templatelinks tables on labsdb1010 for s5 - T174509
  • 07:01 marostegui: Optimize templatelinks and pagelinks tables on s6 codfw master db2028 (this might generate lag) - T174509
  • 06:46 marostegui: Drop now redundant indexes from pagelinks and templatelinks on s7 - T174509
  • 06:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T172679 (duration: 00m 47s)
  • 05:54 marostegui: Optimize templatelinks and pagelinks tables on s5 master codfw (db2023): this will generate lag - T174509
  • 05:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2049 and db2041 - T174509 (duration: 00m 47s)
  • 03:23 demon@tin: Pruned MediaWiki: 1.30.0-wmf.19 [keeping static files] (duration: 01m 30s)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Oct 3 02:30:30 UTC 2017 (duration 6m 39s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 07m 22s)

2017-10-02

  • 23:38 catrope@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/: T177250 (duration: 00m 47s)
  • 23:36 catrope@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FiltersViewModel.js: T177107 (duration: 00m 47s)
  • 23:33 mutante: releases2001 - kill and restart jenkins with systemctl
  • 23:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Update WMF mailing address (T173684) and enable jQuery 3 on Wiktionaries (T124742) (duration: 00m 48s)
  • 23:00 mutante: releases1001 - remove and reinstall jenkins
  • 22:56 Reedy: created newsletter tables on officewiki T176199
  • 19:36 ejegg: turned on CiviMail record creation
  • 18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Change the Turkish Wiktionary logo (T176008) (duration: 00m 46s)
  • 18:46 niharika29@tin: Synchronized static/images/project-logos/: Change the Turkish Wiktionary logo (T176008) (duration: 00m 48s)
  • 17:46 gehel@tin: Finished deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes (duration: 01m 48s)
  • 17:44 gehel@tin: Started deploy [wdqs/wdqs@f20b726]: latest wdqs GUI + minor fixes
  • 14:08 zeljkof: EU SWAT finished
  • 14:07 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Sandbox menu to Javanese Wikipedia (T176308) (duration: 00m 46s)
  • 14:01 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:Newsletter on officewiki (T176199) (duration: 00m 46s)
  • 13:59 zeljkof: extending EU SWAT for 5-10 minutes
  • 13:51 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 45s)
  • 13:50 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
  • 13:44 zfilipin@tin: Synchronized static/images/project-logos/zh_classicalwiki-1.5x.png: static/images/project-logos/zh_classicalwiki-2x.png static/images/project-logos/zh_classicalwiki.png wmf-config/InitialiseSettings.php SWAT: Change the zh-classicalwiki logo (T177165) (duration: 00m 46s)
  • 13:42 godog: roll restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/381045/ - T166699
  • 13:38 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the Extension:SandboxLink on the nlwikinews (T177170) (duration: 00m 46s)
  • 13:32 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 45s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Wikidata: Dont persist description usages (yet) (T177153) (duration: 00m 46s)
  • 13:20 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add eliminator as a priviliged account (T176554) (duration: 00m 46s)
  • 13:17 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 46s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/Wikibase-labs.php: wmf-config/Wikibase.php wmf-config/Wikibase-production.php SWAT: labs: Use redis lock manager for dispatching changes of Wikibase (T175109) (duration: 00m 47s)
  • 12:56 elukey: added two new mediawiki jobrunners - mw131[01]
  • 12:50 marostegui: Optimize tables pagelinks and templatelinks on s5: dbstore2001
  • 11:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2039 - T174509 (duration: 00m 46s)
  • 10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1311.eqiad.wmnet
  • 10:06 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1310.eqiad.wmnet
  • 10:04 mobrovac: restbase restarting on restbase1007 to start logging error bodies for T176728
  • 09:55 jynus: stopping and upgrading labsdb1010
  • 09:27 Amir1: finished the cleanup of ores_classification table in enwiki (T159753)
  • 09:12 godog: roll-restart thumbor to upgrade to 1.6 - T172556
  • 09:06 elukey: forced remount of /mnt/hdfs on notebook1002
  • 08:47 gehel: deploying latest tilerator on maps-test only, fix for T175123
  • 08:46 gehel@tin: Finished deploy [tilerator/deploy@b02bd9a]: (no justification provided) (duration: 00m 22s)
  • 08:46 gehel@tin: Started deploy [tilerator/deploy@b02bd9a]: (no justification provided)
  • 08:46 marostegui: Poweroff db2044 for HW maintenance - T174764
  • 08:40 marostegui: Stop MySQL on db2044 to get it ready to replace its mainboard - T174764
  • 08:29 Amir1: finished the cleanup of ores_classification table in wikidatawiki and starting the enwiki one (T159753)
  • 08:24 jynus: stopping dbstore1001 for maintenance
  • 08:12 marostegui: Optimize tables pagelinks and templatelinks on s5: db1100 (already depooled) - T174509
  • 07:59 marostegui: Drop now redundant indexes from pagelinks and templatelinks from s5 - T174509
  • 07:57 Amir1: starting a round of cleanup in ores_classification table in wikidatawiki
  • 07:41 marostegui: Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311
  • 07:32 marostegui: Optimize table pagelinks and templatelinks on s6: dbstore2001 - T174509
  • 07:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2039 - T174509 (duration: 00m 46s)
  • 07:20 marostegui: Optimize table pagelinks and templatelinks on s6: db2039 - T174509
  • 07:10 marostegui: Optimize table pagelinks and templatelinks on s1: db2034, db2055 db2062 - T174509
  • 07:06 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034, db2055 db2062 - T174509 (duration: 00m 46s)
  • 06:55 _joe_: restarting a few API appservers with high cpu load
  • 06:19 elukey: restart varnish backend on cp3040
  • 06:10 elukey: restart varnish backend on cp3033
  • 06:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)
  • 06:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)
  • 02:43 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 2 02:43:33 UTC 2017 (duration 6m 39s)
  • 02:36 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 08m 42s)

2017-10-01

  • 21:30 madhuvishy: powercycling labvirt1015
  • 18:58 hashar: restarted Jenkins. Was stuck somehow
  • 13:50 elukey: restart hhvm on mw1167 (jobrunner) - hhvm stuck, dump-debug in /tmp/hhvm.9624.bt.

2017-09-30

  • 20:54 catrope@tin: Synchronized wmf-config/CommonSettings.php: Bump $wgResourceLoaderStorageVersion (T176884) (duration: 00m 47s)
  • 00:37 ejegg: increased PayPal Express Checkout cURL timeout to 12 seconds

2017-09-29

  • 22:12 ejegg: updated CiviCRM from 784756b to 59e3e00
  • 21:19 mutante: labstore2003/2004: closed idle screen sessions
  • 20:27 mutante: mwlog1001, oxygen - closing unused screen session
  • 20:20 mutante: install1002/install2002 - closing unused screen sessions (except a tail -f)
  • 20:11 mutante: restbase2001 - closing screen sessions that weren't running anything anymore
  • 19:23 ebernhardson: completed running cirrussearch category moves on commonswiki
  • 18:19 catrope@tin: Synchronized php-1.31.0-wmf.1/includes/libs/CSSMin.php: T176884 (duration: 00m 47s)
  • 18:08 chasemp: remove 'ToolLabs webservice - lighttpd on precise' catchpoint check for Catchpoint
  • 18:07 chasemp: remove labsdb1002 checks on catchpoint (Toolforge)
  • 18:06 chasemp: remove Labs puppetmaster eqiad catchpoint check (for Toolforge)
  • 18:06 chasemp: remove grid-start-precise catchpoint check
  • 18:04 chasemp: update catchpoing test for stream.wikimedia.org to watch https://stream.wikimedia.org/?doc
  • 17:00 ebernhardson: kicking off extra job runners to process jobs moving commonswiki Category pages from general to content serach index
  • 16:37 andrewbogott: re-imaging labvirt1017, 1018 in order to get it on the standard 4.4.0-81-generic kernel
  • 15:52 andrewbogott: re-imaging labvirt1015 in order to get it on the standard 4.4.0-81-generic kernel
  • 13:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2070, db2069 and db2042 - T174509 (duration: 00m 47s)
  • 13:18 Amir1: starting a round of cleanup in ores_classification table in enwiki (T159753)
  • 13:09 elukey: depool mw1265 (hhvm 3.18.5) - disk filled up
  • 12:43 bblack: removed numa=isolate sys entries manually (runtime + future boots) from cp4021
  • 11:53 bblack: cp4021: reboot for numa mode switch
  • 11:44 elukey: re-enable job runner daemons on mw130[8,9]
  • 11:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s)
  • 11:05 elukey: precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation
  • 09:55 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet
  • 09:33 ema: upgrade pybal to 1.14.0 on codfw primaries
  • 08:20 ema: upgrade pybal to 1.14.0 on codfw secondaries
  • 05:53 ema: cp4024 repooled T174891
  • 05:21 ema: powerup cp4024 T174891
  • 00:26 ejegg: updated fundraising tools from 69a9627 to 6d4b6f3
  • 00:16 ejegg: updated CiviCRM 2138869 to 784756b
  • 00:07 mutante: releases1001 - starting Jenkins setup wizard with generated admin pass
  • 00:05 mutante: releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030

2017-09-28

  • 23:46 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s)
  • 23:45 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart
  • 22:18 MaxSem: Deployed fix for T176247
  • 21:30 ejegg: updated CiviCRM from 3f48cb0 to 2138869
  • 21:23 XenoRyet: Updated SmashPig from dc62203 to d056a13
  • 19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
  • 19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798)
  • 19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
  • 19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
  • 19:14 ejegg: updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export
  • 19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
  • 19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
  • 18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
  • 18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
  • 18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
  • 18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
  • 18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
  • 17:58 arlolra: Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832)
  • 17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
  • 17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s)
  • 17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
  • 17:40 moritzm: installing apache updates on cobalt
  • 17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
  • 17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
  • 17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
  • 17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
  • 17:08 bblack: cp4021 - depool->reboot for bios update
  • 17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678]
  • 16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978)
  • 16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
  • 16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
  • 16:01 marostegui: Global rename of CodeCat → Rua - T176985
  • 15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
  • 14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
  • 14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
  • 14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
  • 14:42 marostegui: Remove db1035 from tendril - T176931
  • 14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
  • 14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
  • 13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
  • 13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
  • 13:40 zeljkof: EU SWAT finished
  • 13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
  • 13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
  • 13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
  • 13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
  • 13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
  • 13:12 marostegui: Drop  redundant indexes from pagelinks and templatelinks on s4 - T174509
  • 13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
  • 13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
  • 13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
  • 13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
  • 12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
  • 12:27 ema: reboot cp402[2-6] to disable numa isolation
  • 10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
  • 10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
  • 10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
  • 09:59 elukey: added new mediawiki jobrunner - mw1308
  • 09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
  • 09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
  • 09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
  • 08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
  • 07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
  • 07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
  • 06:59 moritzm: installing apache updates on graphite* hosts
  • 06:28 elukey: restart varnish backend on cp3032
  • 05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
  • 05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
  • 05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
  • 05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
  • 05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
  • 05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
  • 05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
  • 04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
  • 03:57 bblack: ulsfo dns-depooled at ~03:51
  • 03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
  • 03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
  • 02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
  • 00:36 twentyafterfour: upgrading phabricator, expect minimal downtime

2017-09-27

  • 23:39 robh: cp4024 is running hardware testing, leave it alone T174891
  • 23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
  • 22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
  • 19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
  • 19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
  • 19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
  • 18:46 catrope@tin: Finished scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter (duration: 22m 55s)
  • 18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
  • 18:23 catrope@tin: Started scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter
  • 18:21 ejegg: updated payments-wiki from 6104e3d to c2a3d43
  • 17:40 ejegg: updated SmashPig Adyen settings
  • 17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
  • 16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
  • 16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
  • 16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
  • 15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
  • 15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
  • 15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
  • 15:51 ema: lvs4002: upgrade pybal to 1.14.0
  • 15:49 ema: lvs4004: upgrade pybal to 1.14.0
  • 15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
  • 15:24 bblack: rebooting cp402[2356] (bnx2x oops)
  • 15:15 moritzm: revoking salt certs for WMCS
  • 15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
  • 15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
  • 15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
  • 15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
  • 15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
  • 14:58 ema: lvs1007: upgrade pybal to 1.14.0
  • 14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
  • 14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
  • 14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
  • 14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
  • 14:24 mlitn@tin: Started scap: Enable 3D extension
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
  • 14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
  • 14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
  • 14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
  • 14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
  • 14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
  • 13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/. Run puppet as well in a batched execution
  • 13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
  • 13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
  • 13:31 akosiaris: T175242 eventstreams requires manual restart
  • 13:25 akosiaris: T175242 parsoid requires manual restart
  • 13:15 akosiaris: T175242 restbase requires manual restart
  • 13:11 addshore: SWAT done
  • 13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
  • 13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
  • 12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
  • 12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
  • 12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
  • 12:14 moritzm: installing apache updates on einsteinium/tegmen
  • 12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)
  • 12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)
  • 09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
  • 09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
  • 08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
  • 08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
  • 08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
  • 08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
  • 08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
  • 08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
  • 08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:55 marostegui: Optimize pagelinks and template links tables on  db2076 db2067 db2060 - T174509
  • 07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
  • 07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
  • 07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
  • 06:57 marostegui: Deploy alter table on s6 codfw - T174509
  • 06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
  • 06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
  • 06:37 ema: restart varnish-be on cp3040
  • 06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
  • 06:33 ema: restart varnish-be on cp3041
  • 06:29 marosteg1i: Reboot db2044 after storage failure - T174764
  • 06:15 elukey: restart varnish backend on cp3043
  • 05:41 madhuvishy: service nova-fullstack restart on labnet1001
  • 03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
  • 03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)

2017-09-26

  • 23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
  • 23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
  • 23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
  • 22:03 no_justification: gerrit master restart, back momentarily
  • 20:32 mutante: phab1001 - restarted apache after upgrade
  • 20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
  • 20:20 mutante: phab1001 (prod phab) - upgrade Apache package
  • 20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
  • 19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
  • 18:25 mepps: Updated crm from 89b8007 to 3f48cb0
  • 18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
  • 17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
  • 15:50 volans: powecycled cp2012, no ping or ssh console stuck
  • 15:18 XioNoX: starting eqiad frack switch to new infra - T174218
  • 15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
  • 14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
  • 14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
  • 14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
  • 14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)
  • 14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
  • 13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
  • 13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
  • 13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
  • 13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
  • 13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
  • 13:02 marostegui: Drop redundant indexes from s6 - T174509
  • 13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
  • 12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
  • 12:27 marostegui: Drop redundant indexes from s2 - T174509
  • 12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
  • 12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
  • 12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
  • 11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
  • 11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 11:30 kartik@tin: (no justification provided)
  • 11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
  • 11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
  • 10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
  • 10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
  • 10:20 _joe_: rebooting copper (again)
  • 09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
  • 09:10 _joe_: rebooting copper to verify docker setup is correct
  • 08:46 moritzm: installing apache updates on mw* app servers
  • 08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
  • 08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
  • 08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
  • 08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
  • 08:07 marostegui: Drop table trackbacks from s3 - T175051
  • 07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
  • 07:46 marostegui: Drop trackbacks table from s7 - T175051
  • 07:28 marostegui: Drop trackbacks table from s6 - T175051
  • 07:16 moritzm: installing perl security updates
  • 06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
  • 06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
  • 06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
  • 06:18 marostegui: Drop table trackbacks from s4 - T175051
  • 06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
  • 05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
  • 05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)

2017-09-25

  • 23:58 ejegg: updated payments-wiki from d2ce697 to 6104e3d
  • 23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562)
  • 22:53 mutante: gerrit2001 - PXE booting
  • 22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
  • 20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030)
  • 20:37 arlolra: Updated Parsoid to 4230c27b (T176425, T173029)
  • 20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b (duration: 16m 52s)
  • 20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
  • 19:58 XioNoX: renumbering ams BGP communities - T167840
  • 19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
  • 19:02 robh: cp4021 coming down for memory swap
  • 19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
  • 18:54 MaxSem: started populating ip_changes on group2 wikis
  • 18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
  • 18:22 ejegg: updated CiviCRM from 45c56cf to 89b8007
  • 17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
  • 17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
  • 17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
  • 17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
  • 16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
  • 15:57 urandom: T169940: Converting restbase-ng data tables to size-tiered compaction
  • 14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
  • 14:31 gehel: rolling restart of logstash for config change - T176430
  • 14:13 gehel: rolling restart of logstash for config change - T176430
  • 13:46 zeljkof: EU SWAT finished
  • 13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
  • 13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
  • 13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
  • 13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
  • 12:52 moritzm: removed salt master from neodymium
  • 12:38 moritzm: revoked all salt keys on neodymium
  • 11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
  • 11:54 moritzm: removing salt-minion/salt-common from production
  • 11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013)
  • 11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013)
  • 11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572) (duration: 00m 46s)
  • 11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
  • 11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
  • 11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013) (duration: 00m 46s)
  • 11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013) (duration: 00m 45s)
  • 11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013)
  • 11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013) (duration: 00m 46s)
  • 10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
  • 10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
  • 09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
  • 09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
  • 09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
  • 08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
  • 08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
  • 08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
  • 08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
  • 07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
  • 07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
  • 07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
  • 07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
  • 06:38 moritzm: installing apache security updates on logstash*
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)

2017-09-24

  • 14:23 jynus: restarting db2047 for kernel upgrade
  • 14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
  • off: restarted ircecho on einsteinium

2017-09-23

2017-09-22

  • 23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
  • 22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
  • 22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
  • 22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
  • 18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
  • 15:21 hashar: Restarted Jenkins. Out of memory)
  • 14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
  • 14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
  • 14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
  • 13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
  • 12:02 hashar: apt-get upgrade on contint1001 / contint2001
  • 09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
  • 09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
  • 06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)

2017-09-21

  • 23:56 ejegg: updated payments-wiki from 7ed7243 to d2ce697
  • 23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047: Stop delivering search relevance survey javascript (duration: 00m 46s)
  • 23:18 ebernhardson@tin: Synchronized wmf-config: T175047: Turn off human search relevance survey (duration: 00m 48s)
  • 23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047: Turn off human search relevance survey (duration: 00m 46s)
  • 22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
  • 22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 22:10 ejegg: updated payments-wiki from 9da8482 to 7ed7243
  • 21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
  • 21:04 ejegg: updated CiviCRM from fc56901 to 45c56cf
  • 19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
  • 19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
  • 19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
  • 19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
  • 19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
  • 18:56 urandom: T171772: Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
  • 18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
  • 18:41 urandom: T171772: Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
  • 18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
  • 18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
  • 14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
  • 13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
  • 13:12 zeljkof: EU SWAT finished
  • 13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
  • 13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
  • 13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
  • 12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
  • 12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
  • 12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
  • 11:59 moritzm: upgrading apache on deployment servers and script runners
  • 11:40 moritzm: upgrading apache on canary app servers
  • 11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
  • 11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
  • 11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
  • 10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
  • 09:59 dcausse: reindexing group1 wikis T176397
  • 09:54 dcausse: reindexing group0 wikis T176397
  • 08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254)
  • 08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
  • 08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
  • 08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
  • 08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
  • 08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
  • 08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
  • 08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
  • 08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
  • 08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
  • 08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
  • 08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
  • 07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
  • 07:31 _joe_: restarted aphlict on phab1001
  • 07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
  • 07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
  • 07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
  • 06:59 ema: bounce pybal on lvs1006 to clear stale alert
  • 06:55 ema: bounce pybal on lvs1009 to clear stale alert
  • 03:24 bblack: depooled cp4026
  • 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
  • 03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
  • 00:41 eileen: update civicrm from da833ac to fc56901
  • 00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
  • 00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
  • 00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s)
  • 00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s)

2017-09-20

  • 23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
  • 23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
  • 23:40 mutante: phabricator: aphlict (notification service) now running on prod server
  • 23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
  • 23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s)
  • 23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s)
  • 23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
  • 23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s)
  • 23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
  • 22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
  • 21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
  • 21:45 eileen: update civicrm from 0fb4676 to da833ac
  • 21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
  • 20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
  • 20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s)
  • 20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
  • lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
  • 20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263)
  • 20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
  • 19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
  • 19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
  • 19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
  • 19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
  • 19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
  • 19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
  • 19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
  • 19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
  • 19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
  • 19:02 catrope@tin: Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s)
  • 18:40 catrope@tin: Started scap: Patches for T176302, T175962, T173533
  • 17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
  • 17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
  • 17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
  • 17:06 ejegg: removed limit from donation queue consumer
  • 17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
  • 17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
  • 16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
  • 16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
  • 16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
  • 16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
  • 16:34 gehel: cleanup jar hell and restart elastic1051
  • 16:19 gehel: starting elasticsearch on all nodes on eqiad
  • 16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
  • 16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
  • 16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
  • 15:46 ejegg: limited donations queue consumer to 100 per run
  • 15:36 ejegg: updated payments-wiki from 896f16a to 9da8482
  • 15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
  • 15:33 ejegg: limited donations queue consumer to 200 per run
  • 14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
  • 13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
  • 13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
  • 13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 3/3) (duration: 00m 48s)
  • 13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 2/3) (duration: 00m 49s)
  • 13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 1/3, no-op in prod) (duration: 00m 49s)
  • 12:51 ema: restarted varnish-be on cp4028
  • 12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
  • 12:19 ema: restarted varnish-be on cp4027
  • 12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
  • 12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)
  • 11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
  • 11:56 ema: restarted varnish-be on cp4018
  • 11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
  • 11:49 jynus: stopping replication on db1101 for faster repartition work T176311
  • 11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
  • 11:34 ema: restarted varnish-be on cp4010
  • 11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
  • 11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
  • 11:03 moritzm: rolling out new debdeploy release across the fleet
  • 11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
  • 11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:02 ema: restarted varnish-be on cp4008
  • 11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
  • 11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
  • 11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
  • 11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
  • 09:55 bblack: restarted varnish-be on cp3042
  • 09:29 bblack: restarted varnish-be on cp1052
  • 09:23 ema: restarted varnish-be on cp3043
  • 09:21 ema: restarted varnish-be on cp3041
  • 09:19 ema: restarted varnish-be on cp3032
  • 09:14 ema: restarted varnish-be on cp3040
  • 08:42 bblack: restarted varnish-be on cp1053
  • 08:41 bblack: restarted varnish-be on cp1068
  • 08:40 bblack: restarted varnish-be on cp1054
  • 08:40 bblack: restarted varnish-be on cp1065
  • 07:11 ema: cp3030 varnish-be restart due to 503s
  • 03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
  • 00:55 bblack: cp1066 - backend restart, mailbox lag, text
  • 00:54 bblack: cp4016 - backend restart, mailbox lag, text

2017-09-19

  • 23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
  • 23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
  • 23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
  • 23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
  • 23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
  • 23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
  • 23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
  • 23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
  • 23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
  • 22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
  • 22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
  • 22:21 eileen: update civicrm from 9677cf7 to 0fb4676
  • 22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
  • 22:02 mutante: gerrit2001 - systemctl start gerrit
  • 21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
  • 21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
  • 21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
  • 21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
  • 21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
  • 21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
  • 21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
  • 21:06 ema: cp4009: varnish-be restart, 503 spike
  • 20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
  • 20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
  • 19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
  • 19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
  • 19:17 ejegg: updated payments-wiki from 26b16ea to 896f16a
  • 19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
  • 18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
  • 18:35 urandom: T160570, T169940: Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
  • 18:26 arlolra: Updated Parsoid to 05a0965 (T122965, T175101, T173384, T173061, T172896, T174977, T159894)
  • 18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 48s)
  • 18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 49s)
  • 18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 50s)
  • 18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 50s)
  • 18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 12m 26s)
  • 17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
  • 17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
  • 17:43 jynus: stopping db2010 and cloning to db2078
  • 17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
  • 16:45 demon@tin: Started scap: bootstrap wmf.19
  • 16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
  • 16:42 demon@tin: Started scap: bootstrap wmf.19
  • 16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
  • 16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
  • 16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
  • 16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
  • 16:29 jynus: disable puppet on db1009
  • 16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
  • 16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642) (duration: 00m 59s)
  • 16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
  • 16:11 ema: pybal 1.14.0 uploaded to apt.w.o
  • 15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765 (duration: 20m 32s)
  • 15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765
  • 15:22 jynus: enabling firewall on db1020 (m2-master)
  • 15:01 ema: restart varnish-be on cp1055, 503 spike
  • 14:55 jynus: deploying firewall to m5 hosts
  • 14:39 jynus: disabling puppet on all misc active mysql hosts
  • 13:35 zeljkof: EU SWAT finished
  • 13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
  • 13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
  • 13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
  • 13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
  • 12:55 moritzm: rebooting kubestage1001
  • 11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
  • 11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 02m 42s)
  • 11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 10:53 volans: restarted nagios-nrpe-server on stat1005
  • 10:14 jynus: shutting down mysql on db1018
  • 09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
  • 09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
  • 09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
  • 09:30 ema: cp1062 - varnish-backend-restart, mbox lag
  • 08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
  • 08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
  • 08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
  • 08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
  • 07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 00m 37s)
  • 07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 07:50 moritzm: installing bind9 regression update on trusty servers
  • 07:35 kartik@tin: (no justification provided)
  • 07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 01m 04s)
  • 07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
  • 02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
  • 02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
  • 01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
  • 01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
  • 01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
  • 01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
  • 00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
  • 00:10 eileen: update civicrm from 187238f to b6f7dc3

2017-09-18

  • 23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
  • 23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
  • 22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
  • 21:57 bawolff: deployed patch T175900
  • 21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
  • 21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
  • 21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
  • 20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
  • 20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
  • 20:17 arlolra@tin: (no justification provided)
  • 20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 01m 53s)
  • 20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
  • 19:29 XioNoX: restarting pfw3-codfw for software upgrade
  • 19:10 urandom: T160570: Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
  • 19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
  • 19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
  • 19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
  • 19:02 no_justification: gerrit: restarting service, back in a moment
  • 19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
  • 18:56 urandom: T160570: Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
  • 18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
  • 18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
  • 18:37 smalyshev@tin: (no justification provided)
  • 18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
  • 18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
  • 18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
  • 17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
  • 17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
  • 17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
  • 16:46 jynus: shuting down db1100 T175973
  • 16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
  • 16:05 bblack: cp1063 - backend restart, mailbox lag
  • 15:38 bblack: cp1072 - varnish backend restart, mailbox lag
  • 15:28 bblack: cp1074 - backend restart for mailbox lag
  • 15:25 bblack: cp1050 - backend restart for mailbox lahg
  • 14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
  • 13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
  • 13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
  • 13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
  • 13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
  • 13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
  • 13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
  • 12:59 moritzm: installing libxen updates
  • 12:36 moritzm: installing tomcat8 security updates
  • 10:40 ema: cp1099 - backend restart, mailbox lag
  • 06:42 moritzm: installing freexl security updates
  • 06:18 TimStarling: testing EtcdConfig in beta cluster
  • 05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)

2017-09-17

  • 12:49 _joe_: taking a full debug dump of mw1288 after depooling it
  • 12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
  • 09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
  • 08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag

2017-09-16

  • 22:05 godog: compress older otrs directories to reclaim inodes - T171490
  • 16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
  • 15:40 andrewbogott: rebooting labvirt1017
  • 15:34 andrewbogott: rebooting labvirt1015 for T176044

2017-09-15

  • 21:50 mutante: contint2001 - systemctl start zuul
  • 21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
  • 21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
  • 21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
  • 15:46 bblack: cp1049 - backend restart, mailbox lag
  • 15:40 bblack: cp1099 - backend restart, mailbox lag
  • 15:38 bblack: cp1072 - backend restart, mailbox lag
  • 15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
  • 13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
  • 13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
  • 13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
  • 13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
  • 11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
  • 09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
  • 08:54 moritzm: installing libbluetooth updates
  • 08:03 jynus: dropping mariadb analytics users on db1031, db1029
  • 07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601)
  • 07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
  • 07:35 gehel: depooling elastic1020 - T175951
  • 07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
  • 00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)

2017-09-14

  • 23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
  • 23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868) (duration: 00m 47s)
  • 23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
  • 23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
  • 23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
  • 23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
  • 23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095) (duration: 00m 46s)
  • 23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
  • 23:18 mutante: acamar is back up and running - no failed units
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903) (duration: 00m 46s)
  • 23:12 mutante: acamar - staged BIOS upgrade in progress
  • 23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
  • 22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
  • 22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
  • 22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
  • 22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
  • 21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
  • 20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339 (duration: 00m 45s)
  • 20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88 (duration: 00m 46s)
  • 20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df (duration: 00m 46s)
  • 20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
  • 20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
  • 19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
  • 19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
  • 19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
  • 19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
  • 18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
  • 18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
  • 18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
  • 18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
  • 18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
  • 17:57 bblack: cp2001 backend restart, mailbox lag
  • 17:56 bblack: cp1074 backend restart, mailbox lag
  • 17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
  • 17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
  • 17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
  • 16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
  • 15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
  • 15:51 bblack: cp1064 backend restart, mailbox lag
  • 15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
  • 14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898) (duration: 00m 49s)
  • 13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
  • 13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
  • 13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
  • 11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
  • 11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
  • 09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
  • 09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
  • 09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
  • 09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
  • 09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
  • 09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
  • 09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
  • 09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
  • 08:13 gehel: merging cassandra refactoring for puppet - T171704
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
  • 02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
  • 00:46 MaxSem: restarted populateIpChanges to use updated code
  • 00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
  • 00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
  • 00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
  • 00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
  • 00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
  • 00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
  • 00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13

2017-09-13

  • 23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918) (duration: 00m 49s)
  • 23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316) (duration: 00m 57s)
  • 23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
  • 23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413) (duration: 00m 50s)
  • 22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
  • 22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
  • 21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
  • 21:23 ejegg: updated payments-wiki from ed2e481 to 26b16ea
  • grrrr: populating ip_changes on group1 wikis
  • 20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
  • 20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
  • 19:39 andrewbogott: rebooting labtestvirt2001
  • 19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
  • 19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
  • 19:21 urandom: T172384: Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
  • 19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
  • 19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
  • 19:11 urandom: T172384: Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
  • 18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
  • 18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
  • 18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
  • 18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
  • 18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
  • 17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
  • 16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
  • 16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
  • 16:28 gehel: starting decommissioning of wdqs100[12] - T175595
  • 16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
  • 16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
  • 16:22 andrewbogott: rebooting labtestvirt2002
  • 16:08 ottomata: restarting druid-brokers with increase in query cache size
  • 15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
  • 15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
  • 15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
  • 15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
  • 15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038 (duration: 00m 52s)
  • 14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
  • 14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
  • 14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
  • 14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
  • 13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:30 urandom: T169936: Converting RESTBase dev environment to size-tiered compaction
  • 13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
  • 13:23 gehel: initial installation of logstash100[7-9] - T175045
  • 13:21 zeljkof: EU SWAT finished
  • 13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
  • 12:14 moritzm: rebooting mc1015 (spare) for some tests
  • 12:07 jynus: stopping wikidata rebuild maintenance script on terbium
  • 12:07 bblack: cp1049 - backend restart, mailbox lag
  • 11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
  • 11:39 jynus: disabling semi-sync replication on the largest s5 database servers
  • 11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
  • 11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
  • 11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
  • 10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
  • 10:52 volans: reboot lvs3001
  • 10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
  • 09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
  • 09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
  • 08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
  • 08:24 moritzm: installing emacs security updates
  • 07:56 akosiaris: bounce varnish on cp1062, mailbox lag
  • 07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705)
  • 07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
  • 05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36 (duration: 02m 56s)
  • 05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
  • 03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
  • 03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
  • 03:03 jynus: dropping old users from phabricator database
  • 03:00 eileen: update civicrm from ee7dda3 to 187238f
  • 02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
  • 01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
  • 00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)

2017-09-12

  • 23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
  • 23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
  • 22:17 chasemp: labvirt1018:~# /sbin/reboot
  • 22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305) (duration: 04m 28s)
  • 21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305)
  • 21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
  • 21:37 MaxSem: Populating ip_changes in group0 wikis
  • 21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
  • 21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
  • 20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
  • 20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
  • 19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
  • 19:45 demon@tin: Started scap: bootstrap wmf.18
  • 19:05 papaul: maintenance complete on mw2256
  • 17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
  • 17:40 cwd: turned on dedupe
  • 16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
  • 16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
  • 16:40 ejegg: updated CiviCRM from 63288ae to ee7dda3
  • 16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
  • 16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
  • 16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
  • 16:18 ejegg: updated SmashPig from dce4f0a to dc62203910
  • 15:43 _joe_: restarted ircecho on einsteinium
  • 15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
  • 15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
  • 15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
  • 15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
  • 14:53 papaul: shutting down mw2256 for maintenance
  • 14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
  • 14:42 ema: cp1074 - backend restart, mailbox lag
  • 14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
  • 14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
  • 14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
  • 14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
  • 14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
  • 14:06 volans: testing reimage of mw2100 - T166300
  • 14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
  • 14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
  • 14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
  • 13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
  • 13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
  • 13:37 gehel: pooling wdqs100[45] - T171210
  • 13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
  • 13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
  • 13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
  • 13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
  • 13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
  • 13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
  • 13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
  • 09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
  • 09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
  • 09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
  • 08:34 gehel: deploying elasticsearch plugins on relforge - T175159
  • 08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
  • 07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
  • 02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
  • 02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
  • 02:10 bblack: cp1050 - backend restart, mailbox lag
  • 02:08 bblack: cp1099 - backend restart, mailbox lagh
  • 00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
  • 00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart

2017-09-11

  • 23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
  • 22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
  • 22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281, no-op in prod) (duration: 00m 45s)
  • 21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
  • 21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
  • 21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
  • 21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
  • 20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
  • 19:11 chasemp: adjust nodepool rate to 10
  • 19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
  • 19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
  • 18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
  • 18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
  • 18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
  • 18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
  • 18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
  • 18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
  • 18:10 robh: scs-ulsfo has successfully updated firmware T174475
  • 18:03 robh: upgrading firmware on scs-ulsfo
  • 17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
  • 17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
  • 17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
  • 15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
  • 15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
  • 14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
  • 14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
  • 14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
  • 14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
  • 13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
  • 13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
  • 13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
  • 13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
  • 13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
  • 11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
  • 10:30 jynus: restarting mysql @ labsdb1001
  • 09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
  • 09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
  • 09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
  • 08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992 (duration: 00m 07s)
  • 08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992
  • 08:16 volans: reimage script tests continue on mc100[1-2] T166300
  • 07:59 moritzm: installing systemd updates from stretch point update
  • 07:41 moritzm: installing remaining libonig security updates
  • 02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
  • 02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)

2017-09-10

  • 21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
  • 13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
  • 13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
  • 00:37 chasemp: labnodepool1001:~# service nodepool restart
  • 00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues

2017-09-09

  • 23:42 chasemp: labnodepool1001:~# service nodepool start
  • 23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
  • 22:59 chasemp: restart nova-api and nova-network
  • 22:49 chasemp: labnodepool1001:~# service nodepool stop
  • 22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
  • 20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
  • 01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740: Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)

2017-09-08

  • 23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740: Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
  • 23:38 mutante: netmon1002 - terminated unused screen session
  • 23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
  • 21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
  • 20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
  • 20:14 mutante: heze - rebooting for firmware upgrade
  • 20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850)
  • 20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
  • 19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650)
  • 19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650)
  • 18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
  • 16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338, followup (duration: 00m 46s)
  • 16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
  • 15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
  • 15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
  • 15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
  • 15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
  • 15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
  • 14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
  • 12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
  • 12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
  • 10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
  • 10:12 _joe_: stopped all md devices syncs on conf200* machines
  • 10:09 _joe_: etcd cluster lost consensus T162013
  • 10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
  • 09:58 _joe_: running the same command on conf2002 too
  • 09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
  • 09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
  • 09:09 moritzm: installing libonig security updates
  • 09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
  • 09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
  • 09:01 moritzm: installing remaining gnutls updates from jessie point release
  • 08:13 moritzm: installing libgd2 security updates
  • 08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
  • 08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
  • 07:53 moritzm: installing ruby2.3 security updates
  • 07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
  • 07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
  • 07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
  • 07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
  • 00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)

2017-09-07

  • 23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
  • 23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
  • 23:15 XenoRyet: Updated Smashpig from 8eb98c1 to dce4f0a
  • 22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
  • 22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
  • 21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
  • 21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
  • 21:01 urandom: T169940: Disabling puppet and restbase service in RESTBase dev
  • 21:00 urandom: T169940: Disabling changeprop in RESTBase dev environment
  • 19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
  • 18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
  • 18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
  • 18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
  • 17:55 ejegg: disabled queue consumers for data cleaning job
  • 17:52 ejegg: updated CiviCRM from c1ece1e to 63288ae
  • 17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
  • 17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
  • 17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
  • 17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
  • 17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
  • 17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
  • 16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
  • 15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
  • 15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
  • 14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
  • 14:40 jynus: rebooting and upgrading es1019
  • 13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
  • 13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
  • 13:14 zeljkof: EU SWAT finished
  • 13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
  • 13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
  • 11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
  • 09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
  • 08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
  • 08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
  • 08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
  • 07:57 elukey: force re-mount of /mnt/hdfs on stat1005
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
  • 07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
  • 07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
  • 07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
  • 07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
  • 07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
  • 07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
  • 06:07 eileen: civicrm updated from 73a24b4 to c1ece1e
  • 03:13 eileen: seconds behind=78
  • 03:08 eileen: seconds behind 55
  • 02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
  • 02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
  • 02:43 eileen: restarting groupmember.load after hacking out cache delete calls
  • 02:40 eileen: update civicrm from 764bfe1 to 73a24b4
  • 02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
  • 02:05 eileen: replag 217 behind
  • 01:56 eileen: replag 197 seconds
  • 01:47 eileen: replag 23 seconds
  • 01:29 eileen: replag 0
  • 01:22 eileen: replag 550 sec behind master
  • 01:13 eileen: replag delay 422
  • 01:12 eileen: job killed
  • 00:49 eileen: 8389 contacts now
  • 00:49 eileen: 7971 contacts so far
  • 00:46 eileen: started Omnigroupmember.load
  • 00:24 twentyafterfour: phabricator update deployed
  • 00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
  • 00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
  • 00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/

2017-09-06

  • 23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to all webservers refs T174387 (duration: 00m 49s)
  • 23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to wmf.17 refs T174387 (duration: 00m 49s)
  • 23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
  • 23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
  • 23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
  • 23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
  • 23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 Bug: T174106 (duration: 00m 52s)
  • 22:18 no_justification: prior deploy was no-op
  • 22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
  • 22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
  • 22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)
  • 22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)
  • 21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
  • 21:28 ejegg: updated payments-wiki from bf97cd2 to ed2e481
  • 21:24 ejegg: updated CiviCRM from f9c373e to 764bfe1
  • 21:17 ottomata: rebooting kafka-jumbo1004
  • 20:34 arlolra: Updated Parsoid to f9d367ea (T169342)
  • 20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea (duration: 08m 27s)
  • 20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
  • 19:53 ottomata: reimaging kafka-jumbo* with stretch
  • 19:48 chasemp: reboot labvirt1018
  • 18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
  • 18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
  • 18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
  • 18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
  • 18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
  • 18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
  • 18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
  • 18:09 ejegg: rolled payments-wiki back to bf97cd2
  • 18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
  • 18:05 ejegg: updated payments-wiki from bf97cd2 to 6911158
  • 17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
  • 17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
  • 17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156)
  • 17:16 mutante: modify-ldap-group on terbium is broken
  • 17:02 ejegg: disabled creating CiviMail records for Thank You emails
  • 16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa (duration: 01m 01s)
  • 16:41 chasemp: disable puppet for cloud things to test changes
  • 16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
  • 16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
  • 15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
  • 15:53 akosiaris: powercycling lvs3001
  • 15:28 papaul: firmware upgrade on mw2256
  • 15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
  • 15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
  • 15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
  • 14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
  • 13:19 jynus: restarting and upgrading db2040
  • 13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
  • 13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
  • 13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
  • 12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
  • 11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
  • 11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
  • 11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
  • 11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
  • 11:20 godog: reimage restbase1008 with cassandra 3 - T169939
  • 11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
  • 10:26 moritzm: installing perl update from stretch 9.1 point release
  • 10:25 moritzm: installing perl update from jessie 8.9 point release
  • 10:12 godog: reimage restbase1010 with cassandra 3 - T169939
  • 09:39 moritzm: installing libgd security updates
  • 09:28 moritzm: installing libonig security uodates#
  • 09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
  • 08:53 godog: reimage restbase1009 with cassandra 3 - T169939
  • 08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
  • 08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
  • 07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
  • 06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705)
  • 06:37 marostegui: Truncate l10n_cache table across production - T150306
  • 04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
  • 03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
  • 03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
  • 02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
  • away: disabled IPN messages at the PayPal console
  • 00:14 Reedy: that was T174747 Adjust language icon color
  • 00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
  • 00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
  • 00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)

2017-09-05

  • 23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
  • 23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
  • 23:33 foks: 2FA disabled for Omshivaprakash (T175075)
  • 22:17 robh: rebooting mw1163 as its serial console is locked up
  • 22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
  • 22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
  • 20:48 ejegg: re-enabled donation queue consumer and thank you mailer
  • 20:40 ejegg: updated civicrm from 820bd98 to f9c373e
  • 20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
  • 19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
  • 19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
  • 19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
  • 19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
  • 18:11 ejegg: updated fundraising tools from 9554229 to 8d39806
  • 18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
  • 17:38 bblack: cp1074 - restart varnish backend, mailbox lag
  • 17:24 demon@tin: Started scap: wmf.17 bootstrap
  • 17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
  • 16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
  • 16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
  • 16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
  • 16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
  • 16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
  • 16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
  • 16:08 akosiaris: reboot all kubernetes boxes T170119
  • 15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
  • 15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
  • 15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
  • 15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
  • 14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
  • 14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
  • 14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
  • 14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
  • 14:15 zeljkof: EU SWAT finished
  • 14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
  • 13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
  • 13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
  • 13:28 godog: reimage restbase2005 - T169939
  • 13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
  • 13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
  • 12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
  • 12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102) (duration: 00m 46s)
  • 11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
  • 11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 45s)
  • 11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 46s)
  • 10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
  • 10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
  • 10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
  • 10:11 ema: reboot cp4024 T174891
  • 10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
  • 09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
  • 09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
  • 08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705)
  • 08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705)
  • 07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
  • 07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
  • 06:32 marostegui: Deploy alter table on db1083 - T174509
  • 06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
  • 06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)

2017-09-04

  • 23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973)
  • 22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
  • 17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
  • 17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
  • 17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
  • 16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
  • 16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
  • 16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
  • 16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
  • 15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
  • 14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
  • 14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
  • 13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
  • 13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
  • 13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
  • 12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
  • 12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
  • 12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
  • 11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
  • 11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
  • 11:45 godog: restart varnish on cp1099 - mailbox lag
  • 10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
  • 10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
  • 09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
  • 09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
  • 09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
  • 08:52 elukey: restart zookeeper on conf2003 for jvm security updates
  • 08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
  • 08:47 moritzm: installing mercurial security updates
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
  • 08:39 moritzm: installing libgcrypt20 security updates
  • 08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
  • 08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
  • 07:39 moritzm: installing gnupg security updates
  • 07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
  • 07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
  • 04:52 marostegui: Deploy alter table on db1052 - T168661
  • 02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
  • 02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)

2017-09-03

  • 17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
  • 15:57 ema: restart varnish backend on cp1073 (mailbox lag)
  • 09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
  • 09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
  • 09:25 ema: cp4024 is down, powercycled
  • 09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
  • 00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859

2017-09-01

  • 16:18 bblack: restart varnish backend on cp1074 (mailbox lag)
  • {{safesubst:SAL entry|1=15:15 urandom: Restarting Cassandra: restbase-dev100[5-6]-{a,b}}}
  • {{safesubst:SAL entry|1=15:06 urandom: Restarting Cassandra: restbase-dev1004-{a,b}}}
  • 15:00 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s)
  • 14:28 marostegui: Add 150G to /srv partition on labsdb1001
  • 13:42 marostegui: Rename table pr_index on enwiki on db1089 - T174782
  • 13:10 marostegui: Stop MySQL on db1026 as it will be decommissioned - T174763
  • 12:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
  • 12:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
  • 11:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)
  • 11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)
  • 10:35 elukey: stop puppet on thorium and disable root rsyncs - T174756
  • 10:31 marostegui: Upgrade MariaDB to 10.0.32 on db1081 - T168661
  • 10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)
  • 09:10 godog: bounce graphite-web on graphite1001, problematic query? using all CPU
  • 09:04 ema: lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 08:59 godog: depool restbase200[135] before reimage - T169939
  • 07:22 elukey: restart apache2 and hue on thorium, Analytics sites down, investigating
  • 07:06 marostegui: Power reset db2044 as it is unresponsive - T174764
  • 03:54 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s)
  • 00:57 tstarling@tin: Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s)
  • 00:13 RainbowSprinkles: gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so

2017-08-31

  • 20:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
  • 19:27 mattflaschen@tin: Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s)
  • 19:17 cmjohnson1: powercycled mw1294
  • 19:05 mattflaschen@tin: Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030
  • 18:53 mattflaschen@tin: Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s)
  • 18:51 mattflaschen@tin: Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s)
  • 17:50 demon@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s)
  • 16:35 ejegg: re-enabled donation queue consumer and thank you mail sender
  • 15:25 ejegg: updated civicrm from f00611d to 820bd98
  • 15:20 ejegg: updated SmashPig from 8f4388d to 8eb98c1
  • 15:19 ejegg: turned off donation queue consumer and thank you job
  • 15:18 elukey: restart zookeeper on conf200[2,3] for jvm security updates
  • 13:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)
  • 13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)
  • 13:07 elukey: restart zookeeper on conf2001 for security updates (canary node)
  • 13:00 moritzm: installing augeas security updates
  • 12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)
  • 12:14 marostegui: Upgrade MariaDB on db1084 - T168661
  • 12:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)
  • 12:05 akosiaris: reimage chlorine and kubernetes1004 T170119
  • 12:04 akosiaris: reimage chlorine and kubernetes1004 T10119
  • 11:58 moritzm: restarting nginx on dataset1001/ms1001 to pick up libxml security update
  • 11:21 elukey: restart apache2 on bohrium for libxml + gnutls security updates
  • 11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)
  • 11:17 moritzm: restarting apache on dbmonitor* to pick up libxml security update
  • 10:27 moritzm: restarting nginx on prometheus* to pick up libxml security update
  • 10:22 moritzm: restarting apache on einsteinium/tegmen (Icinga hosts) to pick up libxml security update
  • 10:14 moritzm: restarting nginx on meitnerium/archiva.wikimedia.org to pick up libxml security update
  • 10:13 moritzm: restarting nginx on debug proxies to pick up libxml security update
  • 10:10 moritzm: restarting apache on hafnium to pick up libxml security update
  • 09:57 addshore: extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php run done, 6326 hashes recalculated, T172987
  • 09:50 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 # Should upsert 6326 rows T172987
  • 09:44 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 --dry-run # T172987
  • 09:43 moritzm: restarting apache on auth* to pick up libxml security update
  • 09:40 moritzm: installing libxml2 security updates
  • 09:10 moritzm: installing experimental hhvm-luasandbox package on mw1261 (canary app server) with backported patch for T171392 / T173705
  • 09:04 hashar: contint1001 upgrading git, restarting git-daemon and zuul-merger - T161086
  • 09:01 godog: test reimage of restbase2001 - T169939
  • 08:57 hashar: contint2001 upgrading git, restarting git-daemon and zuul-merger - T161086
  • 08:45 godog: remove /srv/deployment/grafana on naos and tin, unused - T170881
  • 08:41 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@eqiad
  • 08:40 addshore@tin: Synchronized php-1.30.0-wmf.16/extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php: T172987 Add waitForReplication to RecalculateCognateNormalizedHashes script (duration: 00m 47s)
  • 08:20 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@codfw
  • 08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
  • 08:01 marostegui: Rename reader_feedback, reader_feedback_history, reader_feedback_pages tables on dewiki on db1092 - T174586
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
  • 06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 with low weight - T168661 (duration: 00m 46s)
  • 06:19 marostegui: Upgrade MariaDB to 10.0.32 on db1056 - T168661
  • 06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
  • 06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
  • 02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 31 02:55:12 UTC 2017 (duration 7m 9s)
  • 02:48 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 06m 40s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 10m 02s)

2017-08-30

  • 23:10 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 47s)
  • 23:09 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 48s)
  • 23:05 demon@tin: Synchronized wmf-config/unitConversionConfig.json: swat (duration: 00m 47s)
  • 21:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.16
  • 21:03 thcipriani@tin: Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: ProofreadPage: Avoids a stack overflow T173520 (duration: 00m 47s)
  • 20:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.15
  • 20:22 ppchelko@tin: Finished deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages (duration: 01m 05s)
  • 20:21 ppchelko@tin: Started deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages
  • 20:07 urandom: T169939: Decommission Cassandra: restbase1009-c.eqiad.wmnet
  • 19:09 ppchelko@tin: Finished deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages (duration: 01m 17s)
  • 19:07 ppchelko@tin: Started deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages
  • 19:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.16
  • 19:04 demon@tin: Synchronized php: Symlink swap for group1/wmf.16 (duration: 00m 46s)
  • 18:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/374399/4 (duration: 00m 47s)
  • 17:29 urandom: T169939: Decommission Cassandra: restbase1009-b.eqiad.wmnet
  • 17:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 17:12 ejegg: disabled creating CiviMail records for Thank You emails
  • 16:49 XioNoX: rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366
  • 16:34 XioNoX: start work on the cr2<->asw-d-codfw link - T174366
  • 16:13 XioNoX: lowering vrrp priority for the cr2<->asw-d-codfw link - T174366
  • 15:56 elukey: re-added analytics1055 among the hdfs/yarn worker after maintenance
  • 15:49 bblack: cp1049 - restart varnish backend, mailbox lag
  • 15:43 urandom: T169939: Decommission Cassandra: restbase1009-a.eqiad.wmnet
  • 15:02 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1294.eqiad.wmnet
  • 14:31 moritzm: restarting nginx on sodium to pick up libxml security update
  • 14:07 elukey: restart java daemons on druid100[456] for jvm security updates
  • 14:04 moritzm: restarting squid on install* servers to pick up libxml security update
  • 14:04 moritzm: restarting squid on installurl downloaders to pick up libxml security update
  • 14:02 moritzm: restarting apache on krypton to pick up libxml security update
  • 13:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1053 original weight - T168661 (duration: 00m 46s)
  • 13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Scale A/B test bucket sizes by 10 - T172291 (duration: 00m 46s)
  • 13:50 moritzm: restarting squid on url downloaders to pick up libxml security update
  • 13:34 moritzm: installing libxml2 security updates
  • 13:23 moritzm: installing ffmpeg security upadtes
  • 13:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 46s)
  • 13:19 hashar: European SWAT completed
  • 13:16 gehel: restarting wdqs-blazegraph and wdqs-updater on all wdqs nodes for logback config change - T172710
  • 13:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on cywiki - T173054 (duration: 00m 48s)
  • 12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 45s)
  • 12:51 gehel: restart wdqs-updater on wdqs2001 for config change
  • 12:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 47s)
  • 12:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 with low weight - T168661 (duration: 00m 46s)
  • 12:05 marostegui: Upgrade MariaDB to 10.0.32 on db1053 - T168661
  • 12:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T168661 (duration: 00m 47s)
  • 11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T174265 (duration: 00m 46s)
  • 10:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1059 - T168661 (duration: 00m 47s)
  • 09:07 elukey: restart all jvm daemons on druid100[123] for security updates
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 48s)
  • 08:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
  • 07:47 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1228.eqiad.wmnet
  • 07:47 marostegui: Upgrade MariaDB on db1059 to 10.0.32 - T168661
  • 07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 for a MariaDB upgrade - T168661 (duration: 00m 53s)
  • 07:14 moritzm: powering down mw1294 for hardware diagnostics (T1674069
  • 07:12 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1294.eqiad.wmnet
  • 07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1055 - T174265 (duration: 00m 52s)
  • 03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 30 03:16:09 UTC 2017 (duration 7m 19s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 15m 42s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 28s)
  • 02:17 demon@tin: Finished deploy [gerrit/gerrit@f33c63f]: Testing, no-op (duration: 00m 07s)
  • 02:17 demon@tin: Started deploy [gerrit/gerrit@f33c63f]: Testing, no-op
  • 01:27 demon@tin: Synchronized wmf-config/InitialiseSettings.php: remove education program from legalteamwiki (duration: 00m 47s)
  • 00:53 demon@tin: Synchronized wmf-config/flaggedrevs.php: killing FR on testwiki (duration: 00m 46s)
  • 00:52 demon@tin: Synchronized dblists/flaggedrevs.dblist: killing FR on testwiki (duration: 00m 47s)
  • 00:05 cwd: turned dedupe off

2017-08-29

  • 23:50 cmjohnson1: reinstalling mw1228
  • 23:34 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Popups/: swat (duration: 00m 47s)
  • 23:33 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: swat (duration: 00m 50s)
  • 21:33 urandom: T169939: Decommission Cassandra: restbase1008-c.eqiad.wmnet
  • 21:16 andrewbogott: restarting designate-sink on labservices1001
  • 21:01 ejegg: updated DjangoBannerStats from 5963e7c to ce95eab
  • 20:56 ejegg: updated payments-wiki from 58d33b8 to bf97cd2
  • 20:49 godog: bounce varnish on cp1074 / cp1099 / cp1072 - mailbox lag
  • 20:38 ppchelko@tin: Finished deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2 (duration: 01m 17s)
  • 20:37 ppchelko@tin: Started deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2
  • 20:36 ppchelko@tin: Finished deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode (duration: 04m 28s)
  • 20:31 ppchelko@tin: Started deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode
  • 19:47 urandom: T169939: Decommission Cassandra: restbase1008-b.eqiad.wmnet
  • 19:34 ottomata: restarting analytics kafka brokers and all mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/
  • 19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.16
  • 19:08 ppchelko@tin: Finished deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config (duration: 06m 48s)
  • 19:02 ottomata: restarting main-eqiad -> analytics kafka mirror maker processes on analytics kafka brokers, something is not working...
  • 19:01 XenoRyet: updated Smashpig from 98c5516 to 8f4388d
  • 19:01 ppchelko@tin: Started deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config
  • 18:19 robh: attempting firmware upgrade on scs-a8-eqiad
  • 18:17 Pchelolo: depooling restbase1008,1009,1010,2003,2005 while cluster reshaping is going on
  • 18:15 demon@tin: Finished scap: bootstrap wmf.16 (duration: 45m 27s)
  • 18:15 urandom: T169939: Decommission Cassandra: restbase1008-a.eqiad.wmnet
  • 17:30 demon@tin: Started scap: bootstrap wmf.16
  • 17:23 ejegg: restarted donation queue consumer and thank you mailer
  • 17:20 Reedy: Disabled oathauth for KartikMistry on wikitech
  • 16:33 ejegg: updated civicrm from 7dfea0d to f00611d
  • 16:32 ejegg: turned off donation queue consumer and thank you mailer
  • 16:04 ejegg: restarted ingenico recurring charge job
  • 15:54 urandom: T169939: Decommission Cassandra: restbase2005-c.codfw.wmnet
  • 14:51 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from db1047 (archived on HDFS) - T172322
  • 14:33 marostegui: Shutdown db1055 to replace its BBU - T174265
  • 13:12 zeljkof: EU SWAT finished
  • 13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
  • 13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
  • 12:37 elukey: restart kafka daemons on kafka1014 for jvm security updates
  • 12:31 godog: bounce pybal on lvs1003 to pick up config changes
  • 11:49 elukey: restart kafka daemons on kafka1013 for jvm security updates
  • 11:43 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1168.eqiad.wmnet
  • 11:30 elukey: restart java daemons on analytics100[1,2] (Hadoop Master nodes) for jvm updates
  • 11:08 marostegui: Stop MariaDB on db1097 to migrate it to file per table - T161088
  • 10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 43s)
  • 10:48 moritzm: installing libxml2 security updates
  • 10:31 moritzm: reimaging mw1168 (video scaler) to jessie
  • 10:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1028 (duration: 00m 42s)
  • 10:15 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1028 (duration: 02m 48s)
  • 09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T168661 (duration: 00m 43s)
  • 09:39 marostegui: Update MariaDB on db1064 to 10.0.32 - T168661
  • 09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 for a mariadb upgrade - T168661 (duration: 00m 43s)
  • 09:33 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1169.eqiad.wmnet
  • 09:29 elukey: re-installed pmacct/librdkafka1/kafkacat on rhenium with stretch versions - T173489
  • 09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T168661 (duration: 00m 43s)
  • 08:53 ema: upgrading cache_text to varnish 4.1.8-1wm1
  • 08:45 akosiaris: reprepro copy calico, calico-cni from jessie-wikimedia to stretch-wikimedia (apt.wikimedia.org) T170119
  • 08:43 hashar: Restarting Jenkins for openjdk update
  • 08:42 elukey: restart yarn/hdfs daemons for openjdk security updates
  • 08:42 akosiaris: upload kubernetes_1.7.4-1 to apt.wikimedia.org/stretch-wikimedia/main T170119
  • 08:42 moritzm: restarting archiva to pick up openjdk security update
  • 08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight - T168661 (duration: 00m 43s)
  • 08:35 gehel: restart wdqs-updater on wdqs2001
  • 08:32 hashar: apt-get upgrade on contint1001 and contint2001
  • 08:20 moritzm: reimaging mw1169 (video scaler) to jessie
  • 08:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight - T168661 (duration: 00m 43s)
  • 08:06 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from dbstore1002 to free space (table archived on HDFS) - T172322 T168303
  • 07:41 marostegui: Upgrade MariaDB on db1091 to 10.0.32 - T168661
  • 07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 42s)
  • 06:56 moritzm: installing ghostscript security updates on trusty (Debian already fixed)
  • 06:09 demon@tin: Synchronized scap/plugins/clean.py: no-op, for consistency (duration: 00m 43s)
  • 06:05 marostegui: Restart MariaDB on db1102 and db1095 to pick up new replication filters - T174385
  • 05:49 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 (duration: 02m 50s)
  • 02:34 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 29 02:34:09 UTC 2017 (duration 6m 44s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 00s)
  • 00:25 urandom: T169939: Decommissioning restbase1010-a.eqiad.wmnet

2017-08-28

  • 23:37 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s)
  • 23:31 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s)
  • 23:27 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s)
  • 23:23 reedy@tin: Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s)
  • 23:22 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s)
  • 23:13 cwd: re-enabled dedupe jobs
  • 23:13 ebernhardson@tin: Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)
  • 23:08 ebernhardson@tin: Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
  • 23:07 ebernhardson@tin: Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
  • 22:56 ejegg: turned fundraising orphan rectifier back on
  • 22:46 urandom: T169939: Decommissioning restbase1010-b.eqiad.wmnet
  • 21:48 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s)
  • 21:37 reedy@tin: Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s)
  • 21:06 ejegg: switched nightly Ingenico audit from wr1 to wx file parsing
  • 21:06 reedy@tin: Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s)
  • 21:01 reedy@tin: Synchronized wmf-config/CommonSettings.php: Move email blacklist to meta:Email_blacklist (duration: 00m 45s)
  • 20:53 XioNoX: pushing "aggregate defaults discard" to all the cr* routers - T174364
  • 20:52 urandom: T169939: Decommissioning restbase1010-c.eqiad.wmnet
  • 20:16 ayounsi@tin: Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s)
  • 20:16 ayounsi@tin: Started deploy [librenms/librenms@5fea59c]: (no justification provided)
  • 20:16 XioNoX: upgrading librenms to 1.31.01
  • 20:08 bd808@tin: Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s)
  • 20:07 bd808@tin: Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes)
  • 19:24 ejegg: updated CiviCRM from 200f34b to 7dfea0d
  • 19:24 ottomata: restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s)
  • 19:15 herron: scb1001: restarted pdfrender service
  • 19:14 ottomata: restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:09 ottomata: rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/
  • 19:03 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s)
  • 18:48 gwicke: restarted pdfrender instances in eqiad (T159922)
  • 18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s)
  • 18:47 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s)
  • 18:47 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
  • 18:43 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s)
  • 18:37 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s)
  • 18:33 XioNoX: upgrading librenms to 1.31
  • 18:32 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s)
  • 18:32 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
  • 18:30 ejegg: updated CiviCRM from c30ec13 to 200f34b
  • 18:21 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s)
  • 18:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s)
  • 18:14 niharika29@tin: Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s)
  • 17:59 XioNoX: pushing "aggregate defaults discard" to *ams - T174364
  • 17:56 urandom: T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet
  • 17:31 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
  • 17:19 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
  • 17:11 XioNoX: pushing "aggregate defaults discard" to cr2-knams - T174364
  • 17:09 gehel@tin: Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s)
  • 17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
  • 17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
  • 16:46 demon@tin: Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s)
  • 16:39 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s)
  • 14:39 elukey: restart eventlogging_sync on dbstore1002 - issue after drop of old table
  • 14:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s)
  • 14:31 elukey: drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720
  • 14:28 zeljkof: EU SWAT finished
  • 14:28 zfilipin@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710) (duration: 00m 45s)
  • 14:01 zeljkof: extending EU SWAT for 10 or so minutes to deploy two more patches
  • 13:55 elukey: restart kafka* daemons on kafka1012 for openjdk security updates (canary)
  • 13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Log WikibaseQualityConstraints (T171281) (duration: 00m 44s)
  • 13:42 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872) (duration: 00m 44s)
  • 13:36 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:33 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to grant/remove transwiki user group in dtywiki (T174226) (duration: 00m 44s)
  • 13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152) (duration: 00m 45s)
  • 13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Remove non-transparent background from dty.wiki logos (T174098) (duration: 00m 45s)
  • 12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
  • 12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet
  • 12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
  • 12:07 gehel: restarting nginx for upgrade on wdqs*
  • 12:05 gehel: restarting nginx for upgrade on elastic* / relforge*
  • 12:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s)
  • 11:05 moritzm: installing libxml2 security updates
  • 08:41 moritzm: restarting squid3 on install1002
  • 08:16 marostegui: Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661
  • 08:10 marostegui: Stop MySQL on db1045 - T172679
  • 08:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s)
  • 07:51 moritzm: reimaging mw1259 to jessie (T145742)
  • 07:22 marostegui: Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265
  • 07:12 moritzm: installing openjdk security updates on notebook* hosts
  • 07:05 moritzm: installing openjdk security updates on meitnerium
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s)

2017-08-27

  • 20:35 ariel@tin: Finished deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete (duration: 00m 02s)
  • 20:34 ariel@tin: Started deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete
  • 15:46 akosiaris: upload kubernetes_1.4.6-7 to apt.wikimedia.org/jessie-wikimedia/main T170346
  • 05:30 marostegui: Force BBU relearn on db1055 - T174265
  • 00:45 jynus: correction last log s/db1051/db1055/
  • 00:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051, hw issues, may get lag (duration: 00m 44s)

2017-08-26

  • 15:46 godog: bounce varnish on cp1073 - mailbox lag

2017-08-25

  • 19:09 gehel: actually not killing the "stuck discovery report-updater process on stat1005", it is already gone - T174110
  • 19:07 gehel: kill stuck discovery report-updater process on stat1005 - T174110
  • 16:03 moritzm: created cn=ciadmin group in LDAP to be used for privileged Jenkins access, initial members in ticket (T169557)
  • 14:29 godog: bounce varnish on cp1049 / cp1072 - mailbox lag
  • 14:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db1097 - T168661 (duration: 00m 44s)
  • 13:53 marostegui: Upgrade MariaDB on db1097 to 10.0.32 - T168661
  • 13:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1097 - T168661 (duration: 00m 44s)
  • 13:40 marostegui: Upgrade MariaDB to 10.0.32 on db2019 - T168661
  • 13:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T168661 (duration: 00m 45s)
  • 13:24 marostegui: Upgrade MariaDB on db2037 to 10.0.32 - T168661
  • 13:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 - T168661 (duration: 00m 44s)
  • 12:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T168661 (duration: 00m 44s)
  • 12:29 marostegui: Upgrade MariaDB on db2044 - T168661
  • 12:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T168661 (duration: 00m 43s)
  • 12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2058 - T168661 (duration: 00m 48s)
  • 11:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 (duration: 00m 47s)
  • 09:19 moritzm: restart thumbor to pick up libxml2 security updates
  • 09:12 godog: reimage restbase2001 to test new partman recipe - T169939
  • 08:41 moritzm: restarting apache/hhvm on canary app servers to pick up libxml security update
  • 08:00 volans: upgrading cumin to v1.0.0 on sarin/neodymium
  • 07:55 gehel: reimaging logstash1006 after change of failed disk - T173679
  • 07:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool db1069 with more weight (duration: 00m 44s)
  • 07:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1033, pool db1069 with low weight (duration: 00m 44s)
  • 06:55 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1033 (duration: 00m 43s)
  • 06:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: formatting fixes (duration: 00m 44s)
  • 06:11 jynus@tin: Synchronized wmf-config/db-codfw.php: repool es2013, formatting fixes (duration: 00m 44s)
  • 06:01 marostegui: Upgrade MariaDB to 10.0.32 on db2058 - T168661
  • 06:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2058 - T168661 (duration: 00m 43s)
  • 05:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T168661 (duration: 00m 44s)
  • 05:18 moritzm: re-enable mw1260 (video scaler) for active job processing
  • 05:16 marostegui: Reboot db2065 to pick up new kernel
  • 05:08 marostegui: Upgrade MariaDB on db2065 - T168661
  • 05:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 for a MariaDB upgrade - T168661 (duration: 00m 44s)
  • 01:12 mattflaschen@tin: Synchronized php-1.30.0-wmf.15/extensions/Flow/modules/engine/components/board/features/flow-board-loadmore.js: T173807: Fix Flow infinite scroll (duration: 00m 45s)

2017-08-24

  • 23:17 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: Disable Ogg Theora video transcodes in default config T172445 (duration: 00m 45s)
  • 22:45 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 [keeping static files] (duration: 02m 01s)
  • 22:24 ejegg: updated civicrm from fa882f2 to c30ec13 (pt-br and ja thank you letters)
  • 21:59 gwicke: restarted pdfrender service instances in eqiad / T159922
  • 21:54 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/373701/1 (duration: 00m 45s)
  • 21:13 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Logging: https://gerrit.wikimedia.org/r/#/c/373691/ (duration: 00m 44s)
  • 20:07 andrewbogott: stopping, starting, restarting etc. nodepool
  • 19:58 urandom: T169939: Decommission Cassandra: restbase2003-c.codfw.wmnet
  • 19:24 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.15
  • 19:14 andrewbogott: restarting rabbitmq-server on labcontrol1001
  • 18:38 ejegg: updated payments-wiki from bd9f730 to 58d33b8
  • 18:38 bblack: varnish backend restart - cp1074 - mailbox lag
  • 18:24 urandom: T169939: Decommission Cassandra: restbase2003-b.codfw.wmnet
  • 17:33 arlolra@tin: Finished deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f (duration: 12m 24s)
  • 17:21 arlolra@tin: Started deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f
  • 17:01 jynus: stopping and cloning db1033 to db1069
  • 16:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give normal weight to db1096 - T172679 (duration: 00m 47s)
  • 16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
  • 16:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
  • 16:34 urandom: T169939: Decommission Cassandra: restbase2003-a.codfw.wmnet
  • 16:24 andrewbogott: apt-get upgrade and updated to MW 1.29.1 on wikitech-static. Rebooting to pick up kernel updates.
  • 15:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some weight to db1096 - T172679 (duration: 00m 47s)
  • 15:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1096 depooled to s5 - T172679 (duration: 00m 47s)
  • 14:44 ladsgroup@tin: Synchronized php-1.30.0-wmf.15/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php: Reduce batch size in WikiPageUpdater (T173710) (duration: 00m 48s)
  • 14:14 phuedx: refreshLinks on bawiki (T173994)
  • 14:13 volans: updated cumin package in our APT to version cumin_1.0.0-1
  • 13:19 phuedx@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T171853: Re-enable Page Previews for enwiki and dewiki on the Beta Cluster (duration: 00m 47s)
  • 13:13 phuedx@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
  • 13:12 phuedx@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 49s)
  • 12:45 gehel: killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
  • 12:30 marostegui: Stop MySQL on db1026 to clone db1096 from it - T172679
  • 12:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 to clone db1096 from it T172679 (duration: 00m 47s)
  • 11:36 jynus: disable puppet on db1069 in preparation for reimage
  • 11:08 marostegui: Global rename Papa1234 → Karl-Heinz Jansen - T173859
  • 09:56 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with full weight (duration: 00m 46s)
  • 09:33 volans: executing https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed to get rid of failed ones
  • 08:54 jynus: 16 minute test run of rebuildTermSqlIndex.php on terbium
  • 08:52 marostegui: Drop tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki - T173590
  • 07:24 Amir1: starting the run for rebuildTermIndex (T171460)
  • 02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 24 02:50:40 UTC 2017 (duration 7m 4s)
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 05m 56s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 09m 50s)
  • 01:46 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue: Deploy 752580637 (T173710) (duration: 00m 49s)

2017-08-23

  • 23:49 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
  • 23:47 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
  • 23:42 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 47s)
  • 23:41 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 49s)
  • 23:28 maxsem@tin: Synchronized php-1.30.0-wmf.15/resources/: https://gerrit.wikimedia.org/r/#/c/373391/ (duration: 00m 48s)
  • 23:27 maxsem@tin: Synchronized wmf-config/MetaContactPages.php: https://gerrit.wikimedia.org/r/#/c/373369/ (duration: 00m 48s)
  • 20:59 urandom: T169939: Truncating MCS tables
  • 20:26 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.15 (duration: 00m 46s)
  • 20:25 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.15
  • 20:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with low weight (duration: 00m 47s)
  • 20:12 thcipriani@tin: Finished scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520 (duration: 04m 11s)
  • 20:08 thcipriani@tin: Started scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520
  • 19:49 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change: pagePreviews: Enable A/B test (BC-only (duration: 00m 47s)
  • 19:45 joal@tin: Finished deploy [analytics/refinery@f467ce1]: Bug fixing deploy (duration: 03m 16s)
  • 19:44 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend: Verify the existence of `url` key when parsing lang objects T172316 (duration: 00m 56s)
  • 19:42 joal@tin: Started deploy [analytics/refinery@f467ce1]: Bug fixing deploy
  • 19:36 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler: Enable WebM playback via ogv.js T172444 (duration: 00m 50s)
  • 19:22 kaldari: foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
  • 19:19 kaldari: mwscript sql.php --wiki=testwiki /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
  • 19:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Remove CitThisPage from blacklist for page previews T173865 (duration: 00m 47s)
  • 19:13 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 49s)
  • 19:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 50s)
  • 19:00 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend/: Verify the existence of key when parsing lang objects https://gerrit.wikimedia.org/r/#/c/373292/ (duration: 00m 49s)
  • 18:57 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Log channel for LoginNotify T173888 (duration: 00m 47s)
  • 18:55 ppchelko@tin: Finished deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains (duration: 01m 33s)
  • 18:54 ppchelko@tin: Started deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains
  • 18:52 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Log everything T173888 (duration: 00m 48s)
  • 18:43 gehel: manually running report updater for discovery golden data on stat1005 - T173333
  • 18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
  • 18:40 gehel: resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
  • 18:39 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
  • 18:32 thcipriani: jenkins restarted, zuul should pick-up queue
  • 18:26 thcipriani: jenkins appears to be having some issues, trying to dump threads now and then restart to see if it helps
  • 18:13 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Allow crats to add people to accountcreator group on mw.or https://gerrit.wikimedia.org/r/#/c/373317/ (duration: 00m 49s)
  • 17:40 bblack: reboot cp4021
  • 16:38 cwd: disabled all dedupe
  • 16:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 [keeping static files] (duration: 01m 32s)
  • 16:24 bd808@tin: Finished deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845) (duration: 00m 39s)
  • 16:23 bd808@tin: Started deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845)
  • 16:22 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 (duration: 03m 44s)
  • 15:35 legoktm: added addshore to "integration" gerrit group (T173233)
  • 15:10 gehel: shutdown logstash1006 for disk replacement - T173679
  • 14:49 moritzm: installing texlive security updates on trusty (Debian already fixed)
  • 14:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4027.ulsfo.wmnet
  • 14:34 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.ulsfo.wmnet
  • 13:31 marostegui: Stop MySQL on db1041 to get it ready for decommission - T173915
  • 13:13 zeljkof: EU SWAT finished
  • 13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow as a Beta feature on wawiki and wawikionary (T172947) (duration: 00m 48s)
  • 12:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
  • 12:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
  • 10:39 jynus: stopping and deleting s1 and s4 from dbstore2001
  • 10:06 joal@tin: Finished deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last) (duration: 03m 30s)
  • 10:03 moritzm: upgrading remaining image scalers to luasandbox 2.0.13
  • 10:02 joal@tin: Started deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last)
  • 09:36 godog: kill older and running tasks from gerrit queue, it was stuck
  • 09:10 moritzm: upgrading remaining API servers in to luasandbox 2.0.13
  • 09:08 godog: upload prometheus-blackbox-exporter 0.7.0+ds1-1~wmf1 to jessie-wikimedia, backported - T169860
  • 08:35 marostegui: Stop Upgrade MySQL on db2073 to 10.0.32 - T168661
  • 08:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T168661 (duration: 00m 59s)
  • 08:18 moritzm: upgrading remaining job runners to luasandbox 2.0.13
  • 07:12 marostegui: Global rename: Opdire657 → Sakiv - T173834
  • 06:55 moritzm: upgrading remaining app servers in to luasandbox 2.0.13
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 23 03:15:36 UTC 2017 (duration 7m 6s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 15m 11s)
  • 02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 03s)

2017-08-22

  • 23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/Linter/includes/LintErrorsPager.php: SWAT: Fix up 11f4a97ba6bcd0c1de (duration: 00m 49s)
  • 23:35 thcipriani@tin: Finished scap: SWAT: Fix typo in the notification message (duration: 21m 48s)
  • 23:13 thcipriani@tin: Started scap: SWAT: Fix typo in the notification message
  • 23:10 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: Remove aborting of BeforePageDisplay hook T173411 (duration: 00m 49s)
  • 21:18 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Retry CodeMirror deployment T170966 (duration: 00m 49s)
  • 20:41 ejegg: updated CiviCRM from 4aa177b to fa882f2
  • 20:25 mutante: restbase-dev* - puppet runs fail due to E: Version '3.11.0' for 'cassandra' was not found
  • 20:00 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.15
  • 19:56 urandom: starting cassandra restbase2004-a and restbase2006-c, OOMs
  • 19:46 thcipriani@tin: Finished scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache (duration: 45m 55s)
  • 19:00 thcipriani@tin: Started scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache
  • 18:50 robh: labservies1002 update completed (not 1003, typo)
  • 18:50 robh: labservies1003 update completed
  • 18:40 robh: firmware update of labservices1002 in progress
  • 17:54 andrewbogott: removing obsolete apache2 and puppetmaster packages from labcontrol boxes for https://phabricator.wikimedia.org/T171786
  • 17:07 thcipriani: starting branch cut for 1.30.0-wmf.15
  • 13:58 godog: bounce varnish on cp1074 / cp1049 / cp1073 - mailbox problems
  • 13:48 zeljkof: EU SWAT finished
  • 13:45 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
  • 13:44 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
  • 13:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseWikimediaShopLink to true for ptwiki (T173768) (duration: 00m 45s)
  • 10:41 moritzm: upgrading hhvm-luasandbox on mw1293-mw1295 (image scalers)
  • 10:21 Amir1: another run of rebuildTermSqlIndex (T171460)
  • 09:54 moritzm: upgrading hhvm-luasandbox on mw1189-mw1208 (API servers)
  • 09:42 marostegui: Renaming user Darwinius → DarwIn - T173159
  • 09:35 moritzm: upgrading hhvm-luasandbox on mw1180-1188 and mw1209-mw1220 (app servers)
  • 09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T172996 (duration: 00m 44s)
  • 08:59 akosiaris: upload apertium-bel-rus_0.2.0~r81186-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:55 moritzm: upgrading hhvm-luasandbox on mw1161-mw1167 (job runners)
  • 08:43 akosiaris: upload apertium-rus_0.1.0~r81184-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:43 akosiaris: upload apertium-bel_0.1.0~r81357-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
  • 08:35 godog: bounce varnish on cp1072 - mailbox problems
  • 08:33 godog: bounce varnish on cp4015 - mailbox problems
  • 08:24 marostegui: Drop s4 from db1095 with NO replication - T172996
  • 08:23 moritzm: upgrading hhvm-luasandbox on deployment servers / script runners
  • 07:34 marostegui: Stop replication on db1064 sanitarium2 and sanitarium3 master to move labsdb1009,10 and 11 s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T172996 (duration: 00m 52s)
  • 07:28 moritzm: installing ruby security updates on trusty (Debian already fixed)
  • 07:11 moritzm: installing c-ares security updates on trusty (Debian already fixed)
  • 06:54 moritzm: installing graphite2 (the image library, not the metrics tool) security updates on trusty (Debian already fixed)
  • 02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 22 02:30:47 UTC 2017 (duration 6m 55s)
  • 02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 07m 11s)

2017-08-21

  • 22:50 robh: rebooting tin for firmware update since its idle
  • 22:20 mutante: praseodymium - installing BIOS upgrade, reboot
  • 22:11 mutante: xenon - installing BIOS upgrade
  • 21:58 mutante: cerium (cassandra test) - rebooting for firmware upgrade
  • 21:54 mutante: cerium - installing Dell BIOS upgrade (T162850)
  • 21:26 mutante: bast2001 - running Dell BIOS firmware upgrade (T162850)
  • 21:20 arlolra@tin: Finished deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b (duration: 10m 59s)
  • 21:09 arlolra@tin: Started deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b
  • 18:40 niharika29@tin: Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)
  • 18:34 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
  • 18:33 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
  • 18:30 niharika29@tin: Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)
  • 18:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Increase AbuseFilter autodisable thresholds for Meta-Wiki T173633 (duration: 00m 44s)
  • 18:22 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add CookBook and Cookbook Talk NS on hiwikibooks T173398 (duration: 00m 45s)
  • 18:16 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Administrators to add/remove 'transwiki' at nowiktionary T172365 (duration: 00m 45s)
  • 18:12 niharika29@tin: Synchronized static/images/: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
  • 18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
  • 17:22 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038 (duration: 00m 14s)
  • 17:22 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038
  • 17:12 gehel@tin: Finished deploy [wdqs/wdqs@a1c4f1f]: (no justification provided) (duration: 02m 27s)
  • 17:10 gehel@tin: Started deploy [wdqs/wdqs@a1c4f1f]: (no justification provided)
  • 15:40 bblack: cp1099 - varnish backend restart for mailbox lag
  • 14:55 mobrovac: cxserver depool scb2001 to debug failed checks - T173038
  • 14:54 mobrovac@tin: Finished deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038 (duration: 00m 18s)
  • 14:53 mobrovac@tin: Started deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038
  • 14:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T163190 (duration: 00m 44s)
  • 14:04 zeljkof: EU SWAT finished
  • 14:04 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
  • 14:02 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
  • 13:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 49s)
  • 13:52 jynus: stop dbstore2001 mariadb@s4, start mariadb@s7
  • 13:52 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 44s)
  • 13:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set X-Frame-Options: SAMEORIGIN if UploadWizard enabled (T173631) (duration: 00m 44s)
  • 13:46 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990
  • 13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 44s)
  • 13:36 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 45s)
  • 13:27 jynus: stop dbstore2001 mariadb@s7
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
  • 13:26 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
  • 13:26 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990
  • 13:11 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
  • 13:10 zfilipin@tin: Synchronized dblists/closed.dblist: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
  • 12:52 marostegui: Stop replication on db1079 and db1041 to compare their data - T163190
  • 12:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T163190 (duration: 00m 45s)
  • 12:35 marostegui: Stop MySQL on db1015 to decommission it - T173570
  • 09:50 jynus: restart dbstore2001 mariadb@s1
  • 09:40 jynus: restart dbstore2001 mariadb@x1
  • 09:29 moritzm: upgrading hhvm-luasandbox on mw1262-1265
  • 09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)
  • 09:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)
  • 09:01 moritzm: upgrading hhvm-luasandbox on mw1261
  • 08:08 marostegui: Rename tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki on db1078 - T173590
  • 07:42 marostegui: Drop cx_drafts table from x1 - T172364
  • 07:13 marostegui: Stop s4 replication thread on db1095 - T172996
  • 07:09 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 (duration: 00m 45s)
  • 02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 21 02:36:58 UTC 2017 (duration 7m 3s)
  • 02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 57s)

2017-08-19

  • 13:17 Amir1: another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井 俊光 → Toshimit) (T173419)
  • 13:00 Amir1: running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
  • 12:47 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
  • 03:21 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on test.wikidata and mediawiki.org - Id4d42d8c53 (duration: 00m 45s)

2017-08-18

  • 21:05 mutante: gerrit2001 - restarted gerrit again after reverting gerrit:372426, using systemctl commands, not 'service' or init.d
  • 20:55 mutante: restarting gerrit on gerrit2001
  • 20:48 demon@tin: Synchronized scap/plugins/clean.py: Completeness (duration: 00m 44s)
  • 20:43 RainbowSprinkles: prior thing was a no-op, testing
  • 20:42 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 43s)
  • 19:03 bblack: cp1074 - varnish backend restart (mailbox lag)
  • 19:01 bblack: reboot cp4021-8
  • 18:21 bblack: puppeting cp4021-8 - expect possibility of ipsec alerts, etc...
  • 13:32 Amir1: another run of /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item --from-id $(tail -100 /tmp/rebuildTermSqlIndex.log | grep -E "Processed up to page (\d+?)" | sed -E "s/Processed up to page //; s/ \(Q.+?//" | tail -1) >>/tmp/rebuildTermSqlIndex.log 2>&1
  • 13:26 Amir1: ladsgroup@terbium:~$ mwscript namespaceDupes.php --wiki=hiwikiversity (T172977)
  • 12:44 Amir1: start of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
  • 11:36 jynus: change topology of dbstore2001:x1 and dbstore2002:x1
  • 11:00 elukey: reboot dbstore1002 for kernel updates
  • 10:34 elukey: restart mysql on dbstore1002 - attempt to reclaim space after big table drop (stop slaves and el_sync, check running queries, stop mysql, check process, start mysql)
  • 09:55 Amir1: one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)
  • 09:46 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)
  • 09:42 moritzm: installing libmspack security updates
  • 09:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2077 - T168409 (duration: 00m 44s)
  • 07:28 marostegui: Stop MySQL on db2077 to copy it to dbstore2001 - T168409
  • 07:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2077 - T168409 (duration: 00m 44s)
  • 07:20 moritzm: installing openjdk-7 security updates on trusty
  • 06:58 jynus: and nginx
  • 06:55 jynus: systemctl restart squid3.service on install2002, apt seem stuck on some servers
  • 06:24 jynus: stopping and upgrading db2033
  • 05:43 jynus: upgrading and restarting all mariadb instances on dbstore2002
  • afk: updated fundraising dash from 696a3ff to 7dfc969
  • 00:48 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (again) to 1k per bucket per day (duration: 00m 44s)
  • 00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (duration: 00m 44s)

2017-08-17

  • 23:31 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Revert "Disable cirrus MLR ab test" (duration: 00m 44s)
  • 23:21 thcipriani@tin: Synchronized php-1.30.0-wmf.14/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FilterGroup.js: SWAT: RCFilters: Fix validation for single_option groups T173303 (duration: 00m 44s)
  • 23:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reapply "Remove temporary wgStructuredChangeFiltersEnableExperimentalViews setting" (duration: 00m 45s)
  • 22:10 ppchelko@tin: Finished deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping (duration: 01m 14s)
  • 22:09 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/specials/SpecialNewpages.php: Restore the newFromId() approach in SpecialNewpages::feedItemDesc T173541 (duration: 00m 46s)
  • 22:09 ppchelko@tin: Started deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping
  • 22:07 gwicke: restarting pdfrender on scb100* nodes
  • 21:38 ejegg: updated dash from bec0077 to 696a3ff
  • 21:28 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify on sites without Echo too. :|https://gerrit.wikimedia.org/r/#/c/372456/ (duration: 00m 44s)
  • 21:19 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify finally! Yippeeeeee https://gerrit.wikimedia.org/r/#/c/372450/ (duration: 00m 45s)
  • 21:17 bblack: cp1063: varnish backend restart (mailbox lag)
  • 20:07 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.14
  • 19:28 chasemp: disable puppet for cloud things for some careful refactor merging
  • 19:14 thcipriani@tin: Synchronized php: group1 wikis to wmf.14 (duration: 00m 46s)
  • 19:12 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to wmf.14
  • 19:07 thcipriani@tin: Finished scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520 (duration: 06m 13s)
  • 19:01 thcipriani@tin: Started scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520
  • 18:28 ebernhardson: restart logstash on logstash1003 to see why its not reading EventError messages from kafka
  • 18:24 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/RevisionSlider/: Revert Reintroduce hover and bar clicking - https://gerrit.wikimedia.org/r/#/c/372384/ (duration: 00m 48s)
  • 18:16 jynus: upgrading and restarting all mariadb instances on dbstore2001
  • 18:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: Log usage statistics https://gerrit.wikimedia.org/r/#/c/372214/ (duration: 00m 51s)
  • 18:03 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/UploadWizard/UploadWizard.config.php: SWAT: Preserve array keys (language keys) when sorting the language dropdown T173522 (duration: 00m 51s)
  • 17:53 jynus: restarting mariadb (dbstore2001:x1) to test new buffer pool configuration
  • 17:20 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520
  • 17:10 RainbowSprinkles: gerrit: cpu spikes, lots of large gc logs. Looking into it. (nb: things might be a little slow, but it is /up/)
  • 16:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173
  • 16:51 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: Avoid lock acquisition errors for multi-title refreshlinks jobs T173462 (duration: 00m 51s)
  • 16:45 ejegg: updated SmashPig from c501f53 to 98c5516
  • 16:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2077 - T170662 (duration: 00m 49s)
  • 16:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2077 - T170662 (duration: 00m 51s)
  • 16:16 urandom: T169939: RESTBase: converting (33) new keyspaces to time-windowed compaction
  • 16:11 urandom: T169939: Lower eqiad compaction throughput from 20MB/s to 15MB/s
  • 15:46 godog: reimage restbase2001 - T169939
  • 15:42 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
  • 14:32 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3036.*
  • 13:30 zeljkof: EU SWAT finished
  • 13:24 zfilipin@tin: Synchronized static/favicon/wikiversity.ico: SWAT: Update wikiversity favicon (T160491) (duration: 00m 50s)
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod to any for srwikiquote (T172974) (duration: 00m 51s)
  • 13:08 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Add one throttling exception (T173444) (duration: 00m 51s)
  • 12:38 moritzm: installing libgd/libsoup security updates
  • 12:08 urandom: T169939: Decommissioning Cassandra/restbase2001-c.codfw.wmnet
  • 12:07 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet
  • 10:29 gilles: Thumbor stress test finished
  • 10:20 gilles: Load testing Thumbor
  • 10:13 godog: roll-restart nginx on thumbor to apply https://gerrit.wikimedia.org/r/372199
  • 08:36 godog: roll-restart thumbor to apply webp support change
  • 08:30 godog: restart varnish on cp1062 to fix mailbox lag
  • 08:05 godog: reboot ms-be2021, unreachable on network
  • 06:24 marostegui: Stop slave on db2047 to fix duplicate keys - T151029
  • 05:55 marostegui: Stop replication on db2077 to fix duplicate entries - T151029
  • 05:48 marostegui: Stop replication in sync on db1078 and db1015 - T164488
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 17 03:11:13 UTC 2017 (duration 7m 18s)
  • 03:03 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 03m 58s)
  • 02:50 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 33s)
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 52s)
  • 00:45 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet

2017-08-16

  • 22:30 urandom: T169939: Decommissioning Cassandra/restbase2001-a.codfw.wmnet
  • 22:14 urandom: T169939: Cleaning up wikipedia parsoid snapshots
  • 22:13 urandom: T169939: Rolling restart of Cassandra complete
  • 21:38 bawolff: deleting private info from securepoll_votes that the script missed due to ref-integ issues for old elections (T173393)
  • 21:37 thcipriani: train is on hold pending resolution of T173462
  • 21:33 bawolff: deleting private info in enwiki arbcom1_vote table (T173393)
  • 21:30 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack d
  • 21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462
  • 21:21 thcipriani@tin: Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)
  • 20:46 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack b
  • 20:45 arlolra@tin: Finished deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e (duration: 08m 52s)
  • 20:37 arlolra@tin: Started deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e
  • 19:57 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack a
  • 19:52 MaxSem: Manually cleaning up PI on enwiki (T173393)
  • 19:37 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.14 (duration: 00m 46s)
  • 19:35 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.14
  • 19:06 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack d
  • 18:57 thcipriani@tin: Synchronized wmf-config/CirrusSearch-common.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART II (duration: 00m 50s)
  • 18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART I (duration: 00m 51s)
  • 18:41 thcipriani@tin: Synchronized wmf-config: SWAT: Apply token count limits to phrase queries on all wikis T172653 (duration: 00m 53s)
  • 18:30 thcipriani@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 50s)
  • 18:29 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 51s)
  • 18:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable group1 T163380 (duration: 00m 54s)
  • 18:18 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack c
  • 17:31 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack b
  • 17:07 demon@tin: Synchronized php-1.30.0-wmf.14/extensions/AntiSpoof/SpoofUser.php: T173394 (duration: 00m 51s)
  • 15:54 marostegui: Stop MySQL and shutdown db1078 for HW checks - T173365
  • 15:43 elukey: drop PageContentSaveComplete_5588433_15423246 from db1047 and dbstore1002 (analytics-slaves)
  • 15:41 godog: delete outdated CFs cassandra metrics from graphite2002 and graphite1003
  • 15:35 marostegui: Rename cx_drafts table on db1029 - T172364
  • 15:09 godog: restart varnish on cp1072 to clear mailbox lag
  • 14:56 godog: restart varnish on cp1099 to clear mailbox lag
  • 14:48 godog: restart varnish on cp1074 to clear mailbox lag
  • 14:44 godog: restart varnish on cp1049 to clear mailbox lag
  • 14:38 gilles: Thumbor stress test finished
  • 14:29 gilles: Stress testing Thumbor from single IP
  • 13:57 zeljkof: EU SWAT finished
  • 13:35 zfilipin@tin: Synchronized dblists/pp_stage1.dblist: SWAT: pagePreviews: Deploy to next 100 stage 1 wikis (T162672) (duration: 00m 50s)
  • 13:09 zfilipin@tin: Synchronized portals: (no justification provided) (duration: 00m 52s)
  • 13:08 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 52s)
  • 12:25 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)
  • 12:25 moritzm: powercycling cp3036
  • 12:24 marostegui: Compressing InnoDB on db2077 - T168409
  • 12:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: cp3036.esams.wmnet
  • 10:46 marostegui: Stop replication in sync on db1015 and db1078 - T164488
  • 10:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2076 - T170662 (duration: 00m 51s)
  • 10:03 godog: copy ubuntu-font-family-sources to stretch-wikimedia - T170817
  • 08:37 marostegui: Drop wikigrok tables from s1, s3 and s5 - T172020
  • 08:12 marostegui: Stop MySQL on db2047 to copy its content to db2077 - T170662
  • 08:10 moritzm: bounced pdfrender on scb1004 (T159922)
  • 08:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T170662 (duration: 01m 06s)
  • 07:25 marostegui: Stop MySQL on db2076 to copy its content to dbstore2001 - T168409
  • 07:08 elukey: executed sudo find -type f -mtime +30 -exec rm {} \; in /var/log/carbon to free some space
  • 06:40 marostegui: Run pt-table-checksum on s3 for revision table - T164488
  • 06:01 marostegui: Stop replication on db2076 to fix duplicate entries - T151029
  • 03:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 16 03:35:43 UTC 2017 (duration 7m 21s)
  • 03:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 16m 32s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 08s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 51s)
  • 00:43 dereckson@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372088) (duration: 00m 51s)
  • 00:42 dereckson@tin: Synchronized php-1.30.0-wmf.14/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372051) (duration: 00m 51s)
  • 00:38 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on three French wikis (T154371) + Fixes for Wikidata: Remove wbq_evaluation logging, Update Wikidata property blacklist (Gerrit:367913 and Gerrit:370846) (duration: 00m 53s)
  • 00:02 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on test wikis (T173388) (duration: 00m 51s)

2017-08-15

  • 23:58 dereckson@tin: Synchronized php-1.30.0-wmf.14/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 50s)
  • 23:55 dereckson@tin: Synchronized php-1.30.0-wmf.13/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 54s)
  • 23:14 Dereckson: Ran updateArticleCount.php on sr.wikiquote (T172974)
  • 23:12 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for hiwikiversity (T172977) (duration: 00m 53s)
  • 22:50 ppchelko@tin: Started restart [changeprop/deploy@444223d]: (no justification provided)
  • 22:45 Pchelolo: scb - codfw: enabling puppet and starting changeprop back T169939
  • 21:40 mobrovac: scb - codfw: disabling puppet and stopping changeprop to release load on Cassandra while dropping old tables - T169939
  • 21:26 urandom: Starting Cassandra restbase2005-b
  • 19:24 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14
  • 19:19 mobrovac@tin: Finished deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939 (duration: 02m 50s)
  • 19:16 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939
  • 19:14 demon@tin: Finished scap: bootstrap wmf.14 v2.0 (duration: 30m 56s)
  • 19:04 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones) - T169939
  • 19:03 mobrovac@tin: Finished deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939 (duration: 04m 39s)
  • 18:58 mobrovac@tin: Started deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939
  • 18:43 demon@tin: Started scap: bootstrap wmf.14 v2.0
  • 18:39 demon@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3982832257" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 05s)
  • 18:35 demon@tin: Started scap: bootstrap wmf.14
  • 18:18 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 [keeping static files] (duration: 02m 47s)
  • 17:59 bsitzmann@tin: Finished deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362) (duration: 04m 00s)
  • 17:55 bsitzmann@tin: Started deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362)
  • 15:21 mobrovac: restarting pdfrender on scb1001, added some debug messages to help us diagnose T159922
  • 13:42 gehel: cleanup of old leftover logs on deployment-logstash2:/var/log/logstash
  • 13:21 moritzm: bounced pdfrender on scb1001 (T159922)
  • 11:32 moritzm: installing Linux updates on stretch systems (no reboots yet)
  • 10:45 moritzm: installing PHP security updates on trusty
  • 10:09 jynus: rebooting db1078
  • 09:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1078 (duration: 00m 49s)
  • 07:56 moritzm: installing cvs security updates
  • 07:18 elukey: restart pdfrender on scb1003
  • 02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 15 02:56:14 UTC 2017 (duration 6m 42s)
  • 02:49 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 53s)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 58s)

2017-08-14

  • 22:16 Dereckson: Previous log entry is related to Gerrit:371960 and Gerrit:371961.
  • 22:15 dereckson@tin: Synchronized php-1.30.0-wmf.11/extensions/EventBus/JobQueueEventBus.php: JobQueueEventBus: Populate the database field + not set properties are accessed (duration: 00m 48s)
  • 20:44 ppchelko@tin: Finished deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation (duration: 09m 35s)
  • 20:35 ppchelko@tin: Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation
  • 17:09 gehel: resetting mode on stat1005:/srv/published-datasets/discovery recursively
  • 16:29 elukey: execute sudo find -type f -mtime +60 -exec rm {} \; in /var/log/carbon on graphite2001 to free some space in /
  • 13:32 elukey: Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681
  • 13:08 zeljkof: nothing for EU SWAT
  • 12:21 jynus: stopping replication on all instances of dbstore2001 T169516
  • 11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s)
  • 11:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s)
  • 10:34 marostegui: Restart MySQL on db1069 to pick up new replcation filters
  • 10:27 marostegui: Restart MySQL on db1095 to pick up new replication filters
  • 09:16 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet
  • 09:07 _joe_: stopping thumbor on thumbor2001 after depooling it for testing
  • 09:05 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s)
  • 02:39 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s)

2017-08-13

  • 22:45 ebernhardson: restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152
  • 19:36 godog: upload python-thumbor-wikimedia 1.2 - T161719
  • 19:01 godog: bounce pdfrender on scb1001 and scb1003 - T159922
  • 18:46 godog: bounce carbon and uwsgi on graphite1003
  • 17:53 reedy@tin: Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s)
  • 17:52 reedy@tin: Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s)
  • 17:51 reedy@tin: Synchronized refresh-dblist: phpcs (duration: 00m 48s)

2017-08-12

  • 20:00 krinkle@tin: Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)
  • 16:09 Reedy: Deleted some bogus user languages from commonswiki.user_properties
  • 15:25 elukey: powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346

2017-08-11

  • 21:47 legoktm@tin: Finished scap: (no justification provided) (duration: 07m 09s)
  • 21:41 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet
  • 21:40 legoktm@tin: Started scap: (no justification provided)
  • 21:40 legoktm@tin: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s)
  • 21:02 ebernhardson: unban elastic1017 from elasticsearch cluster
  • 20:52 bblack: varnish backend restart on cp1099
  • 20:46 legoktm@tin: Started scap: Deploying Timeless (try 2) - T154371
  • 20:42 legoktm@tin: scap aborted: Deploying Timeless - T154371 (duration: 05m 10s)
  • 20:36 legoktm@tin: Started scap: Deploying Timeless - T154371
  • 20:19 bblack: varnish backend restart on cp1049 + cp1074 (mailbox lag)
  • 19:59 ebernhardson: ban elastic1017 from eqiad search cluster
  • 18:28 moritzm: installing subversion security updates
  • 17:19 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s)
  • 17:05 twentyafterfour: Deploying phabricator security update
  • 16:21 jynus_: stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki
  • 15:41 jynus_: stopping db2075 to clone it to dbstore2001
  • 15:39 moritzm: installing git security updates on trusty (jessie/stretch already fixed)
  • 15:38 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
  • 13:51 elukey: moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506
  • 13:49 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s)
  • 13:49 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:42 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s)
  • 13:42 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:25 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s)
  • 13:24 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:16 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
  • 13:15 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 13:00 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 17s)
  • 13:00 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:51 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 04s)
  • 12:51 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:44 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 05s)
  • 12:44 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
  • 12:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T151029 (duration: 00m 48s)
  • 11:25 jynus: stopping and upgrading labsdb1010
  • 11:19 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
  • 09:28 jynus: stopping and restarting es2013 for upgrade
  • 09:07 jynus: stopping and restarting db2046 for upgrade
  • 07:35 elukey: restart pdfrender on scb1004

2017-08-10

  • 23:43 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/371196/2 (duration: 00m 47s)
  • 23:26 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 47s)
  • 23:25 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 46s)
  • 22:31 twentyafterfour@tin: Synchronized php-1.30.0-wmf.13/includes/specials/SpecialDoubleRedirects.php: Hopefully fix T173045 (duration: 00m 48s)
  • 22:17 moritzm: installing git security-updates
  • 20:51 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371 (duration: 00m 21s)
  • 20:51 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371
  • 20:50 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371 (duration: 00m 09s)
  • 20:50 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371
  • 20:35 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371 (duration: 00m 05s)
  • 20:34 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371
  • 20:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371 (duration: 18m 17s)
  • 19:42 mobrovac@tin: Started deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371
  • 19:35 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 47s)
  • 19:33 reedy@tin: Synchronized wmf-config/: Newsletter on testwiki (duration: 00m 49s)
  • 19:31 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340 (duration: 01m 08s)
  • 19:30 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340
  • 19:30 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340 (duration: 00m 34s)
  • 19:29 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340
  • 19:28 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340 (duration: 00m 12s)
  • 19:28 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340
  • 19:27 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: All wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
  • 19:16 godog: restart cassandra instances on xenon to test logstash-logback-encoder deploy
  • 19:13 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340 (duration: 00m 03s)
  • 19:13 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340
  • 18:56 Reedy: created newsletter tables on testwiki
  • 18:55 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/Newsletter/: sql file updates (duration: 00m 52s)
  • 18:53 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaMaintenance/createExtensionTables.php: newsletter (duration: 00m 52s)
  • 18:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Fix commons/commonswiki snafu (duration: 00m 52s)
  • 17:53 kartik@tin: Finished deploy [cxserver/deploy@f43ef96]: (no justification provided) (duration: 00m 44s)
  • 17:52 kartik@tin: Started deploy [cxserver/deploy@f43ef96]: (no justification provided)
  • 17:42 kartik@tin: Finished deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3 (duration: 00m 40s)
  • 17:42 kartik@tin: Started deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3
  • 16:23 ema: cp1072: restart varnish backend
  • 15:48 XioNoX: troubleshoting interface errors between pfw3-codfw and fasw-codfw
  • 15:09 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
  • 15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T151029 (duration: 00m 50s)
  • 15:03 marostegui: Poweroff es2013 for maintenance - T172265
  • 14:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2075 - T170662 (duration: 00m 51s)
  • 14:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T151029 (duration: 00m 51s)
  • 14:11 elukey: restart kafka1012 temporary with some logs to TRACE to debug T172681
  • away: updated civicrm from 200abc2 to 4aa177b
  • 13:43 marostegui: Compress cebwiki on db1095 - T153058
  • 13:32 zeljkof: EU SWAT finished
  • 13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod to any on srwikisource (T172974) (duration: 00m 53s)
  • 13:18 zeljkof: EU SWAT, part two
  • 13:17 marostegui: Drop m3 databases from dbstore1002 - T156758
  • 13:16 zeljkof: EU SWAT finished
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on knwiki (T172894) (duration: 00m 52s)
  • 12:56 marostegui: Stop MySQL on db2045 to upgrade socket location - T148507
  • 12:07 elukey: restored varnishakafka on cp3032
  • 11:17 elukey: disabled puppet on cp3032 and restarted varnishkafka with debug logging
  • 09:58 jynus: continuing cloning of dbstore2002 to dbstore2001
  • 09:30 gehel: repooling wdqs2001, long after data reload completed
  • 09:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
  • 09:27 gehel@tin: Finished deploy [wdqs/wdqs@c186e3e]: (no justification provided) (duration: 01m 31s)
  • 09:25 gehel@tin: Started deploy [wdqs/wdqs@c186e3e]: (no justification provided)
  • 09:17 jynus: disabling lag notification on all s4 replicas
  • 08:59 elukey: update librdkafka1 to 0.9.4.1 on eventlog1001
  • 08:15 elukey: add 50G to carbon lv on graphite1003 and 100G on graphite2002
  • 07:32 jynus: enabling semisymc master replication on db1068
  • 07:29 jynus: disabling semisync slave replication on all SSD hosts on s4
  • 06:45 elukey: powercycle mw2256 - T163346
  • 06:38 elukey: restart pdfrender on scb1004
  • 06:04 Dereckson: Removed 2FA for GoldRingChip account (T172878)
  • 03:23 bblack: cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn"
  • 03:20 bblack: batched cumin puppet agent run on all hosts (not forced)
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s)

2017-08-09

  • 22:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s)
  • 21:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s)
  • 21:11 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s)
  • 21:05 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s)
  • 21:01 godog: add 100G to carbon lv on graphite1003 and graphite2002
  • 20:58 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
  • 20:54 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s)
  • 20:51 marostegui: Remove m3 replication from dbstore1002 - T156758
  • 20:47 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy
  • 20:43 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s)
  • 20:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s)
  • 20:35 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3
  • 20:22 twentyafterfour: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
  • 18:57 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2
  • 18:39 marostegui: Stop replication on db2045 to fix duplicate keys - T151029
  • 18:36 urandom: Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness)
  • 18:27 urandom: Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness)
  • 18:10 ebernhardson: restart es on elastic1018 to test interleaved numa
  • 17:50 marostegui: Add db2076 to tendril - T170662
  • 17:41 marostegui: Compress innodb on s6 on db2076 - T170662
  • 17:31 legoktm: changed mediawiki/vendor to fast forward only
  • 17:24 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s)
  • 17:23 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
  • 16:35 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s)
  • 16:30 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation
  • 16:29 urandom: T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal)
  • 16:15 marostegui: Stop mysql on db2046 to copy its content to db2076 - T170662
  • 16:14 elukey: rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793
  • 16:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s)
  • 16:08 gehel: config update on logstash, a few logs might be lost during restart - T172713
  • 15:19 XioNoX: removing old pfw related config from cr1-codfw - T171970
  • 14:08 mutante: purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404
  • 14:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s)
  • 14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s)
  • 13:55 marostegui: Reboot db2076 for maintenance - T170662
  • 13:21 marostegui: Stop replication on db2075 for maintenance - T170662
  • 13:18 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s)
  • 13:10 gehel@tin: Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s)
  • 13:09 gehel@tin: Started deploy [wdqs/wdqs@6620e0f]: (no justification provided)
  • 12:41 gehel: restarting updater on wdqs1001 (real fix coming up soon
  • 10:20 jynus: stopping dbstore2002's mysqls and cloning them to dbstore2001
  • 08:39 jynus: disable puppet on dbstore2001, about to be reimaged
  • 07:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s)
  • 07:12 jynus: stopping replication in sync between db1069 and db1065, db1044, db1064, db1070
  • 07:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s)
  • 06:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s)
  • 06:11 jynus: stopping db1069 (s7) and db1079 replication in sync
  • 06:03 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085, depool db1079 (duration: 00m 51s)
  • 05:42 jynus: stopping db1069 (s6) and db1085 replication in sync
  • 05:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060, depool db1085 (duration: 00m 50s)
  • 04:56 jynus: stopping db1069 (s2) and db1060 replication in sync
  • 04:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 58s)
  • 04:48 jynus: restarting pdfrender on scb1001, unresponsive
  • 04:09 jynus: powercycling mw2256
  • 03:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 33s)
  • 02:56 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 45s)
  • 02:38 ebernhardson: restart elasticsearch1017 with niofs store
  • 02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 38s)
  • 00:27 ebernhardson: test vm.zone_reclaim_mode=0 on elastic1017

2017-08-08

  • 23:23 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats on WMF wikis to grant and remove 'confirmed' (T101983) (duration: 00m 51s)
  • 21:25 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.13 refs T170631
  • 21:08 ppchelko@tin: Finished deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API (duration: 08m 31s)
  • 21:05 twentyafterfour@tin: Finished scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631 (duration: 42m 55s)
  • 20:59 ppchelko@tin: Started deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API
  • 20:32 twentyafterfour: updated mediawiki.org changelog for 1.30.0-wmf.13
  • 20:22 twentyafterfour@tin: Started scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
  • 19:46 ema: restart varnish backend on cp1074
  • 19:45 godog: bounce thumbor-instances on thumbor1003 to make sure all memory limits are applied
  • 19:44 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="test2wiki" --outdir="/tmp/scap_l10n_224168097" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 07m 33s)
  • 19:36 twentyafterfour@tin: Started scap: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
  • 18:28 XioNoX: frack-codfw moved to new infrastructure
  • 18:27 elukey: re-enabled irc-echo after the puppet shower
  • 18:11 elukey: stop ircecho to avoid puppet shower
  • 16:59 twentyafterfour: Branching mediawiki/master to mediawiki/wmf/1.30.0-wmf.13 refs T170631
  • 16:04 elukey: rolling restart of varnishkafka-webrequest to apply https://gerrit.wikimedia.org/r/#/c/370659/ (puppet automatically restarts)
  • 15:03 XioNoX: starting pfw-codfw migration - T171970
  • 14:19 elukey: restart of all the varnishkafka statsv/eventlogging instances on caching hosts to pick up https://gerrit.wikimedia.org/r/370644 (puppet automatic restarts)
  • 14:16 elukey: set mw2256 pooled=inactive + downtime to allow BIOS upgrade - T163346
  • 14:07 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add copyright info for Wikidata API (T112606) (duration: 00m 47s)
  • 13:56 gehel: dump of task API for elasticsearch eqiad - T169498
  • 13:54 ladsgroup@tin: Synchronized static/images/project-logos: SWAT: Fix srwiki logos (T150618) (duration: 00m 48s)
  • 13:00 elukey: restart varnishkafka-webrequest with kafka.broker.version.fallback=0.9.0.1 + kafka.api.version.request=false on cp3032 (local test, to rollback remove the lines from /etc/varnishkafka/webrequest.conf)
  • 12:49 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property --rebuild-all-terms (T172776)
  • 12:32 elukey: restart pdfrender on scb1002
  • 12:20 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T172776)
  • 12:15 gehel: restarting wdqs-updater on wdqs1001
  • 12:14 elukey: stop eventlogging on eventlog1001 to test kafka consumer failures
  • 11:22 Amir1: start of ladsgroup@terbium:~$ timeout 3500s /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item >>/tmp/rebuildTermSqlIndex.log 2>&1 (T171460)
  • 10:12 elukey: update librdkafka1* on notebook100[12] and stat1003
  • 09:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 as the main recentchanges/watchlist s6 role (duration: 00m 47s)
  • 08:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 at 50% load (duration: 00m 46s)
  • 08:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 with limited load (duration: 00m 46s)
  • 07:53 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1098 (duration: 00m 47s)
  • 07:41 elukey: stop puppet on cp3032 (cache::text) to set varnishkafka-webrequest logging to debug
  • 07:34 Amir1: stopped the script and re-running without --deduplicate-terms (T171460)
  • 07:21 Amir1: start of ladsgroup@terbium:~$ time mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --entity-type=property --deduplicate-terms (T171460)
  • 06:12 elukey: alert users with big home directories for stat1005 disk alarms (will erase data later on only if they don't answer)
  • 06:12 elukey: restart pdfrender on scb1003
  • 03:55 mutante: phab1001 /usr/local/bin/community_metrics.sh | /usr/local/bin/project_changes.sh creating stats mails to admins (which failed before) (T163938)
  • 03:25 mutante: phab1001 /srv/phab/tools/public_task_dump.py to create dump, was failed cron due to missing /srv/dumps/
  • 02:53 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 8 02:53:13 UTC 2017 (duration 6m 53s)
  • 02:46 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 06m 59s)
  • 02:36 ejegg: enabled donation queue consumer
  • 02:26 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 25s)
  • 00:46 ejegg: disabled donation queue consumer

2017-08-07

  • 21:28 urandom: T172384: Upgrading Cassandra to 3.11.0-wmf1 in dev environment (build patched to disable in-built heap dumping)
  • 19:51 ejegg: updated CiviCRM from f24ba78 to 200abc2
  • 19:05 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212 - Turn on CirrusSearch MLR AB test (duration: 00m 46s)
  • 19:00 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 2) (duration: 00m 46s)
  • 18:59 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 1) (duration: 00m 46s)
  • 18:53 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: T171212 - Update CirrusSearch AB test rescore profiles (duration: 00m 46s)
  • 18:44 ebernhardson@tin: Synchronized wmf-config/Wikibase-labs.php: T112606 - beta only - Add copyright info for Wikidata API (duration: 00m 46s)
  • 18:43 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172630 - Enable wgMinervaEnableSiteNotice for kowiki (duration: 00m 46s)
  • 18:38 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)
  • 18:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CirrusSearch/: T169498 limit phrase token count, T172464 constant boost ltr queries (duration: 00m 58s)
  • 18:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172594 - Translate sitename for nl.wikinews (duration: 00m 47s)
  • 18:05 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Grant autopatrol to editor in en.wikibooks - T172561 (duration: 00m 47s)
  • 17:57 mutante: phab2001 - re-enabling puppet, but closing firewall for 80/443
  • 17:56 jynus: stopping slave and reparitioning db1098
  • 17:10 marostegui: Restart s7 instance on db1102 to pick up new replication filters - T172693
  • 17:05 gehel@tin: Finished deploy [wdqs/wdqs@da33919]: (no justification provided) (duration: 02m 28s)
  • 17:02 gehel@tin: Started deploy [wdqs/wdqs@da33919]: (no justification provided)
  • 16:44 marostegui: Restart s7 instance on db1069 to pick up new replication filters - T172693
  • 16:37 XioNoX: manually restarted varnish on cp1099
  • 15:10 thcipriani: restarting jenkins for plugin upgrade
  • 14:51 gehel: reducing elasticsearch eqiad concurrent rebalance to 4 (from 8)
  • 14:38 elukey: updated librdkafka1 and ++1 to 0.9.4.1 on hafnium
  • 14:32 mutante: phab2001 - stopping Apache,schedule downtime for http and puppet
  • 14:22 herron: mx[1,2]001, fermium: Installed libmail-dkim-perl and restarted spamassassin service - T172689
  • 13:15 jynus: reboot db1098
  • 12:39 _joe_: restarting pdfrender on scb1001, T159922
  • 12:39 elukey: restart kafka on kafka1018 to force it out of the kafka topic leaders - T172681
  • 12:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2074 - T171321 (duration: 00m 45s)
  • 12:08 gehel: deploying https://gerrit.wikimedia.org/r/#/c/299825/ - some logs will be lost during logstash restart
  • 10:02 marostegui: Add dbstore2002:3313 to tendril - T171321
  • 09:47 jynus: stopping db1050's mysql and cloning it to db1089
  • 09:06 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 (was 120) on all the analytics kafka brokers - T136094
  • 09:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 after fixing: linter, page and watchlist tables (duration: 00m 47s)
  • 08:12 marostegui: Force BBU re-learn on db1016 - T166344
  • 07:02 marostegui: Stop replication on db2065 to reimport: page, linter and watchlist tables
  • 07:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 to reimport: page, linter and watchlist tables (duration: 00m 47s)
  • 06:38 marostegui: Stop MySQL on db2074 - T171321
  • 06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2074 - T171321 (duration: 00m 46s)
  • 06:33 marostegui: Stop replication on db2075 - T170662
  • 06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2073 - T171321 (duration: 00m 47s)
  • 06:20 marostegui: Force BBU re-learn on db1016 - T166344
  • 02:57 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 7 02:57:42 UTC 2017 (duration 6m 42s)
  • 02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 56s)
  • 02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 10m 16s)

2017-08-06

  • 13:17 elukey: powercycle mw2256 - com2 frozen - T163346
  • 13:13 elukey: restart pdfrender on scb1002
  • 06:18 ebernhardson@tin: Synchronized wmf-config/PoolCounterSettings.php: T169498: Reduce cirrus search pool counter to 200 parallel requests cluster wide (duration: 02m 54s)
  • 01:28 chasemp: conf2002:~# service etcdmirror-conftool-eqiad-wmnet restart (not sure what else to do the service failed)

2017-08-05

  • 14:40 Reedy: created oauth tables on foundationwiki T172591
  • 14:13 reedy@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaMaintenance/createExtensionTables.php: add oauth (duration: 00m 48s)

2017-08-04

  • 23:51 mutante: phab2001 - removed outdated /etc/hosts entries, that fixed rsync, syncing /srv/repos/ from phab1001
  • 23:35 mutante: phab2001 rebooting
  • 23:35 mutante: phab2001 - installing various package upgrades, apt-get autoremove old kernel images
  • 23:12 mutante: "reserved" UID 498 for phd on https://wikitech.wikimedia.org/wiki/UID | phab2001: find -exec chown to fix all the files , restart cron
  • 23:04 mutante: phab2001 - changing UID/GID for phd user from 997:997 to 498:498 to make it match phab1001, to fix rsync breaking permissions. (rsync forces --numeric-ids when fetching from and rsyncd configured with chroot=yes). chown -R phw:www-data /srv/repos/
  • 22:37 ejegg: restarted donations and refund queue consumers
  • 21:44 ejegg: stopped donations and refund queue consumers
  • 21:24 urandom: T172384: Disabling Puppet in dev environment to prevent unattended Cassandra restarts
  • 20:19 mutante: renewing SSL cert for status.wm.org (just like wikitech-static, but that one didnt have monitoring?)
  • 20:02 mutante: wikitech-static-ord - apt-get install certbot
  • 19:41 ejegg: updated CiviCRM from f1fd7f0 to f24ba78
  • 19:38 mutante: renaming graphite varnish director/fixing config, running puppet on cache misc, tested on cp1045
  • 18:17 andrewbogott: switched most cloud instance to new puppetmasters, as per https://phabricator.wikimedia.org/T171786
  • 11:46 marostegui: Deploy schema change directly on s3 master for maiwikimedia - T172485
  • 11:30 marostegui: Deploy schema change directly on s3 master for kbpwiki - T172485
  • 11:14 marostegui: Deploy schema change directly on s3 master for dinwiki - T172485
  • 10:14 marostegui: Deploy schema change directly on s3 master for atjwiki - T172485
  • 10:05 marostegui: Stop replication on db2073 for maintenance
  • 09:22 marostegui: Add dbstore2002 to tendril - T171321
  • 09:19 marostegui: Deploy schema change directly on s3 master for techconductwiki - T172485
  • 08:35 marostegui: Deploy schema change directly on s3 master for hiwikiversity - T172485
  • 08:19 marostegui: Deploy schema change directly on s3 master for wikimania2018wiki - T172485
  • 08:04 marostegui: Sanitize wikimania2018wiki on sanitarium and sanitarium2 - T155041
  • 07:47 marostegui: Stop MySQL on db2073 to copy its data to dbstore2002 - https://phabricator.wikimedia.org/T171321
  • 07:47 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T171321 (duration: 00m 47s)
  • 07:07 moritzm: installing imagemagick regression security updates on trusty
  • 06:47 marostegui: Sanitize hiwikiversity on sanitarium and sanitarium2 - T171829
  • 05:23 mutante: phab1001 sudo ip addr del 10.64.32.186/32 dev eth0 (T172478)
  • 02:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet
  • 02:28 dzahn@neodymium: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
  • 02:15 mutante: phab1001 can't talk to mx servers via IPv6, but works via IPv4. iridium and other mailservers can also talk IPv6 to it. why? it did not change even when stopping ferm on client and on server it allows from anywhere. workaround for now was to hardcode IPv4 IP in phab config. (T163938)
  • 02:13 twentyafterfour: outgoing phab mail is working again
  • 01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chown -R phd:www-data /srv/repos
  • 01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chmod -x /usr/local/sbin/sync-srv-repos
  • 01:12 mutante: phab1001 - stopped and started exim, which is now running with same options as iridium
  • 00:53 twentyafterfour: starting phd to shut up icinga
  • 00:40 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn CirrusSearch MLR test back off (duration: 00m 47s)
  • 00:35 twentyafterfour: phabricator web service is also up
  • 00:34 mutante: phab1001 - service IPs switched - puppet ran - ssh-phab service up
  • 00:30 twentyafterfour: phab1001 chown -R phd:phd /srv/repos
  • 00:28 twentyafterfour: testing phd on phab1001
  • 00:24 mutante: phab1001 sudo ip addr add 2620:0:861:103:10:64:32:186/128 dev eth0
  • 00:24 mutante: phab1001 sudo ip addr add 10.64.32.186/32 dev eth0
  • 00:23 mutante: iridium sudo ip addr del 2620:0:861:ed1a::3:16/128 dev lo
  • 00:23 mutante: iridiium sudo ip addr del 208.80.154.250/32 dev lo
  • 00:23 mutante: iridium sudo ip addr del 2620:0:861:103:10:64:32:186/128 dev eth0
  • 00:23 mutante: iridium sudo ip addr del 10.64.32.186/32 dev eth0
  • 00:07 twentyafterfour: stoped phd and apache on iridium
  • 00:07 twentyafterfour: stopped ssh-phab on iridium
  • 00:03 mutante: phab1001 - /usr/bin/rsync -av rsync://iridium.eqiad.wmnet/srv-repos /srv/repos/
  • 00:02 twentyafterfour: Taking phabricator down for maintenance / migration to a new server: phab1001

2017-08-03

  • 23:43 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212: Turn on CirrusSearch MLR AB test (duration: 00m 46s)
  • 23:36 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370117/2 (duration: 00m 47s)
  • 23:14 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror/resources/ext.CodeMirror.js: Only show popup if CodeMirror button exists (duration: 00m 46s)
  • 23:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 47s)
  • 23:08 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch config for MLR test (step 2) (duration: 00m 46s)
  • 23:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: CirrusSearch config for MLR test (step 1) (duration: 00m 47s)
  • 22:06 ejegg: restarted donation and refund queue consumers
  • 21:51 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370096/ (duration: 00m 47s)
  • 21:51 ejegg: enabled ingenico_audit drupal module
  • 21:45 ejegg: updated civicrm from 5c741b1 to f1fd7f0
  • 21:43 ejegg: disabled donation and refund queue consumers
  • 21:31 chasemp: clear out nova-fullstack project so we can monitor fresh on new puppetmaster
  • 21:30 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Enable CodeMirror https://gerrit.wikimedia.org/r/#/c/370072/ (duration: 00m 47s)
  • 21:20 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370066/ https://gerrit.wikimedia.org/r/#/c/370074/ (duration: 00m 47s)
  • 21:04 herron: copper /var/cache/pbuilder 95% full - grew /dev/copper-vg/pbuilder fs by +5G and tune2fs -m 0. now at 85% full
  • 20:40 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config (duration: 11m 51s)
  • 20:29 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis except wikidata to 1.30.0-wmf.12, leaving wikidata behind due to T172320
  • 20:28 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config
  • 20:24 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script (duration: 33m 17s)
  • 19:51 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script
  • 19:21 twentyafterfour@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/extensions/Wikibase/client/includes/Changes: be sure https://gerrit.wikimedia.org/r/#/c/369847/ is sync'd (duration: 00m 46s)
  • 19:12 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs T168053
  • 19:08 twentyafterfour: deploying 1.30.0-wmf.12 to group1, will proceed to group2 after verifying the branch is stable.
  • 18:59 arlolra: Updated Parsoid to 6e1a20d5 (T168765, T155038, T165977)
  • 18:53 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Final gerrit 369960: "pagePreviews: Deploy to first 50 of stage 1 wikis" (duration: 00m 46s)
  • 18:52 arlolra@tin: Finished deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5 (duration: 06m 25s)
  • 18:51 demon@tin: Synchronized wmf-config/CommonSettings.php: new dblist for gerrit 369960 (duration: 00m 46s)
  • 18:46 arlolra@tin: Started deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5
  • 18:41 demon@tin: Synchronized docroot/: gerrit 369960 (duration: 00m 47s)
  • 18:39 demon@tin: Synchronized dblists/: gerrit 369960 (duration: 00m 47s)
  • 18:29 demon@tin: Synchronized wmf-config/InitialiseSettings.php: last part of gerrit 351287 (duration: 00m 47s)
  • 18:23 mobrovac@tin: Finished deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765 (duration: 08m 33s)
  • 18:17 demon@tin: Synchronized wmf-config/CommonSettings.php: Add new dblist for page preview stuff, basically no-op (duration: 00m 47s)
  • 18:15 demon@tin: Synchronized docroot/noc/conf/pp_stage0.dblist: tidy up pp config (duration: 00m 46s)
  • 18:15 mobrovac@tin: Started deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765
  • 18:14 demon@tin: Synchronized dblists/pp_stage0.dblist: clean up config (duration: 00m 47s)
  • 18:11 mobrovac@tin: Finished deploy [restbase/deploy@65af18d] (staging): (no justification provided) (duration: 01m 57s)
  • 18:09 mobrovac@tin: Started deploy [restbase/deploy@65af18d] (staging): (no justification provided)
  • 18:03 arlolra: Updated Parsoid to 651f12c2 (T119802, T73386, T170289)
  • 17:55 andrewbogott: restarting rabbitmq-server on labcontrol1001
  • 17:50 arlolra@tin: Finished deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2 (duration: 11m 17s)
  • 17:39 arlolra@tin: Started deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2
  • 16:15 jynus: repointing dbproxy1008 back to db1043
  • 16:06 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: This is a test, disregard (duration: 00m 03s)
  • 16:05 demon@tin: Started deploy [gerrit/gerrit@15f1544]: This is a test, disregard
  • 16:03 demon@tin: Synchronized wmf-config/: doc fixes, no-op (duration: 00m 48s)
  • 16:02 demon@tin: Synchronized scap/plugins/prep.py: doc fixes, no-op (duration: 00m 47s)
  • 16:00 godog: silence cassandra-related alerts on restbase-dev cluster, known OOMs
  • 15:44 urandom: T172384: lower tombstone failure threshold in RESTBase dev to 1000
  • 15:39 reedy@tin: Synchronized wmf-config/interwiki.php: Update for 3 new wikis (duration: 00m 46s)
  • 15:31 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: techconductwiki
  • 15:27 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: techconductwiki (duration: 00m 46s)
  • 15:25 reedy@tin: Synchronized static/images/project-logos/: techconductwiki (duration: 00m 44s)
  • 15:24 reedy@tin: Synchronized dblists/: techconductwiki (duration: 00m 47s)
  • 15:03 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic1028.eqiad.wmnet
  • 15:02 gehel: unbanning and repooling elastic1028 - T168816
  • 14:58 reedy@tin: Synchronized static/images/project-logos/: hiwikiversity (duration: 00m 46s)
  • 14:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hiwikiversity (duration: 00m 48s)
  • 14:55 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hiwikiversity
  • 14:55 reedy@tin: Synchronized dblists: hiwikiversity (duration: 00m 49s)
  • 14:53 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(29|30|31|32).eqiad.wmnet
  • 14:53 Dereckson: mwscript initSiteStats.php --wiki srwikiquote --update (T172241)
  • 14:52 gehel: unbanning and repooling elastic10(29|30|31|32) - T168816
  • 14:29 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikimania2018wiki (duration: 00m 47s)
  • 14:28 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
  • 14:27 reedy@tin: Synchronized dblists: wikimania2018wiki (duration: 00m 47s)
  • 14:21 marostegui: Restart MySQL on db2019 to get the new socket location updated
  • 14:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T170662 (duration: 00m 46s)
  • 14:06 marostegui: Compress s5 on db2075 - T170662
  • 14:02 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(28|29|30|31|32).eqiad.wmnet
  • 14:01 gehel: depooling and shutting down elastic10(28|29|30|31|32) - T168816
  • 14:00 marostegui: Add db2075 to tendril - T170662
  • 13:44 marostegui: Enable gtid back on codfw s4 slaves - T170351
  • 13:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Promote db2051 as s4 codfw master - T170351 (duration: 00m 46s)
  • 13:28 elukey: drop CookieBlock backup tables for T171883
  • 13:26 marostegui: Starting the actual s4 codfw failover db2019 -> db2051 - T170351
  • 13:22 gehel: banning elastic1032 - T168816
  • 13:19 chasemp: disabling puppet acros wmcs things for testing
  • 13:14 marostegui: Start topology change for s4 in codfw, slaves will be moved under db2051 - T170351
  • 13:10 aude@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: Fix bug in InjectRCRecordsJob (duration: 02m 12s)
  • 13:01 marostegui: Disable gtid on s4 codfw slaves to get ready for the topology change - T170351
  • 12:15 marostegui: Restart MySQL on db2051 - T170351
  • 10:21 elukey: removed /run/pacct_shadow.d on stat1005 to allow atopacct.service restart
  • 10:01 moritzm: installing poppler security updates on trusty
  • 09:57 _joe_: systemctl reset-failed puppetmaster.service on labpuppetmaster1002
  • 09:52 moritzm: installing mysql 5.5 security updates (package as shipped by Debian jessie, not our internal wmf-mariadb package)
  • 09:32 gehel: banning elastic103[01] - T168816
  • 08:56 ema: upgrading cache_upload to varnish 4.1.8-1wm1
  • 08:36 gehel: banning elastic102[89] - T168816
  • 08:35 marostegui: Stop MySQL on db2045 to copy its data to db2075 - T170662
  • 08:34 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2045 - T170662 (duration: 00m 47s)
  • 08:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db2045 - T170662 (duration: 00m 47s)
  • 07:58 marostegui: Add db2073 and db2074 to tendril - T170662
  • 06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
  • 06:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
  • 06:01 marostegui: Compress s7 on db1102 - T172169
  • 05:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2051 - T170351 (duration: 00m 54s)
  • 05:27 marostegui: Deploy alter table on enwiki - labsdb1010 - T166204
  • 05:17 marostegui: Stop replication on labsdb1011 for maintenance - T153743
  • 03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 3 03:09:22 UTC 2017 (duration 7m 18s)
  • 03:02 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 47s)
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 53s)
  • 01:40 twentyafterfour: repositories synced for phabricator migration.
  • 00:28 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Simply ores (duration: 00m 47s)
  • 00:26 twentyafterfour: scheduled downtime for phabricator migration
  • 00:12 mutante: phab1001 starting ferm service
  • 00:11 mutante: iridium restarted ferm - rsync fragment was in config but not applied, breaking data rsync to phab1001
  • 00:05 mutante: rsyncing /srv/repos from iridium to phab1001 (T163938)

2017-08-02

  • 23:53 ebernhardson@tin: Finished scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages (duration: 36m 21s)
  • 23:17 ebernhardson@tin: Started scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages
  • 21:48 ejegg: updated SmashPig from f4ca53c to c501f53
  • 20:24 bsitzmann@tin: Finished deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325) (duration: 05m 38s)
  • 20:19 bsitzmann@tin: Started deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325)
  • 19:50 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.11 refs T168053 - rollback due to T172320
  • 19:24 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs refs T168053
  • 19:23 twentyafterfour: group1 wikis to 1.30.0-wmf.12 refs T168053
  • 18:41 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialRecentchanges.php: SWAT: Follow-up 31be7d0: send tags list if experimental mode is disabled (duration: 00m 47s)
  • 18:21 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta-only change Enable HTML5 sections in betalabs (duration: 00m 46s)
  • 18:20 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(24|25|26|27).eqiad.wmnet
  • 18:18 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialUndelete.php: SWAT: Fix Special:Undelete search - use variable and not request param (duration: 00m 46s)
  • 18:18 gehel: un-banning and repooling elastic102[4567] - T168816
  • 18:13 thcipriani@tin: Synchronized wmf-config/jobqueue.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part II (duration: 00m 47s)
  • 18:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part I (duration: 00m 47s)
  • 17:56 ottomata: restart kafka1012 broker with listeners=PLAINTEXT://:9092 to verify https://gerrit.wikimedia.org/r/#/c/356232/ before merge. This should be a functional no-op
  • 17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2 (duration: 00m 16s)
  • 17:44 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2
  • 17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963 (duration: 00m 29s)
  • 17:43 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963
  • 17:26 mobrovac@tin: Finished deploy [recommendation-api/deploy@8dfae34]: (no justification provided) (duration: 01m 36s)
  • 17:25 mobrovac@tin: Started deploy [recommendation-api/deploy@8dfae34]: (no justification provided)
  • 17:18 herron: installed libmail-spf-perl and restarted spamassassin service on mx[1,2]001 - T172299
  • 16:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(24|25|26|27).eqiad.wmnet
  • 16:43 gehel: depooling and shutting down elastic102[4567] - T168816
  • 15:34 mobrovac@tin: Finished deploy [cxserver/deploy@cf3e280]: (no justification provided) (duration: 00m 22s)
  • 15:34 mobrovac@tin: Started deploy [cxserver/deploy@cf3e280]: (no justification provided)
  • 15:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2051 IP - T169501 (duration: 00m 46s)
  • 15:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2051 IP - T169501 (duration: 00m 46s)
  • 15:17 nschaaf@tin: Finished deploy [recommendation-api/deploy@baa11c0]: source parameter validation (duration: 03m 22s)
  • 15:14 nschaaf@tin: Started deploy [recommendation-api/deploy@baa11c0]: source parameter validation
  • 15:06 marostegui: Poweroff db2051 to get it move to another rack - T169501
  • 14:46 ema: upgrading cache_misc to varnish 4.1.8-1wm1
  • 14:31 ema: varnish 4.1.8-1wm1 (fixes VSV00001, DSA 3924-1) built and uploaded to apt.w.o
  • 14:12 marostegui: Stop MySQL on db2051 in order to get it ready to move to another rack - T170351
  • 13:16 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Write to term_full_entity_id column in wb_terms table in prod too - T167229 (duration: 00m 46s)
  • 13:07 hashar@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: fix constraint type checks - T169326 (duration: 02m 13s)
  • 12:59 gehel: banning elastic102[67] - T168816
  • 12:41 gehel: banning elastic102[45] - T168816
  • 11:42 jynus: disabling autolearn on newest db1* hosts
  • 11:39 jynus: disabling autolearn on newest db2* hosts
  • 11:33 marostegui: Stop MySQL on db2051 for maintenance - T170351
  • 11:29 jynus: setting all es hosts with disabled auto-learn bbu properties
  • 11:22 kartik@tin: (no justification provided)
  • 11:21 kartik@tin: (no justification provided)
  • 11:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2051 - T170351 (duration: 00m 46s)
  • 11:18 kartik@tin: Finished deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7 (duration: 00m 56s)
  • 11:17 kartik@tin: Started deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7
  • 11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool es2013 (duration: 00m 47s)
  • 11:02 hashar: Upgraded tox on CI to 2.5.0
  • 10:49 marostegui: Force a re-learn cycle on es2013
  • 10:18 marostegui: Rename wikigrok tables on enwiki on db1089 - T172020
  • 09:03 marostegui: Drop table click_tracking_user_properties wherever it exists - T115982
  • 09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2057 - T170662 (duration: 00m 46s)
  • 08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2073 to s4 - T170662 (duration: 00m 47s)
  • 08:50 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2073 to s4 - T170662 (duration: 00m 57s)
  • 08:32 marostegui: Drop table click_tracking wherever it exists - T115982
  • 08:15 godog: bounce pdfrender on scb1001, stuck
  • 07:09 marostegui: Disable BBU auto-learn on es2013
  • 06:58 marostegui: Stop MySQL on db2057 for maintenance - T148507
  • 06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T166204 (duration: 00m 45s)
  • 05:47 demon@tin: Synchronized dblists/: No-op (duration: 00m 47s)
  • 02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 2 02:59:33 UTC 2017 (duration 7m 3s)
  • 02:52 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 54s)
  • 02:36 eileen: update process-control to 3d3978a
  • 02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 35s)

2017-08-01

  • 23:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: Make Babel use databasey stuff T145366 (duration: 00m 46s)
  • 23:46 eileen: update process control to33a0262d87aa1512888c82ac6b4ed91de3d3cac5
  • 23:44 reedy@tin: Synchronized wmf-config/CommonSettings.php: wfLoadExtension Scribunto (duration: 00m 46s)
  • 23:43 reedy@tin: Synchronized wmf-config/extension-list: json! (duration: 00m 46s)
  • 23:41 reedy@tin: Synchronized wmf-config/CommonSettings.php: Unbreak ContactPage T172199 (duration: 00m 46s)
  • 23:39 reedy@tin: Synchronized wmf-config/MetaContactPages.php: Unbreak ContactPage T172199 (duration: 00m 46s)
  • 23:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: T167071 (duration: 00m 47s)
  • 23:30 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikinews - T172211 (duration: 00m 46s)
  • 23:26 reedy@tin: Synchronized php-1.30.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.Controller.js: T172156 T171514 (duration: 00m 46s)
  • 23:24 reedy@tin: Synchronized static/images/project-logos/: optimise pngs (duration: 00m 46s)
  • 23:23 eileen: update process_control to ecb6669
  • 23:23 reedy@tin: Synchronized static/apple-touch/: optimise pngs (duration: 00m 46s)
  • 23:21 reedy@tin: Synchronized docroot/noc/css/images/: optimise pngs (duration: 00m 47s)
  • 23:13 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable popups on Special pages T170893 (duration: 00m 47s)
  • 23:09 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: New wordmarks T168203 T171769 (duration: 00m 47s)
  • 23:08 reedy@tin: Synchronized static/images/mobile/copyright/: 2 new workmarks (duration: 00m 48s)
  • 22:57 chasemp: push procs off swap for labcontrol1001 w/ swapoff -a & swapon -a
  • 22:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/369434/3 (duration: 00m 47s)
  • 22:20 eileen: tools updated from 58bcbf3 to 9554229
  • 22:17 eileen: update process_control to c198b7f
  • 21:48 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy x2
  • 21:47 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: Initial deploy (duration: 00m 40s)
  • 21:47 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy
  • 21:46 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
  • 21:45 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: (no justification provided) (duration: 00m 06s)
  • 21:45 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
  • 21:44 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
  • 20:40 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.12 refs refs T168053
  • 19:33 twentyafterfour@tin: Finished scap: Sync 1.30.0-wmf.12 and build l10n refs T168053 (duration: 29m 58s)
  • 19:19 urandom: restarting Cassandra, restbase1004-a.eqiad.wmnet, aberrant read latency
  • 19:03 twentyafterfour@tin: Started scap: Sync 1.30.0-wmf.12 and build l10n refs T168053
  • 18:49 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 02s)
  • 18:49 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
  • 18:47 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 16s)
  • 18:47 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
  • 18:43 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: (no justification provided) (duration: 00m 48s)
  • 18:42 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: (no justification provided)
  • 17:23 twentyafterfour: MediaWiki train for 1.30.0-wmf.12 - finished `scap prep` & `scap patch` refs T168053
  • 16:41 ejegg: updated CiviCRM from 23f2bbf to 5c741b1
  • 16:25 twentyafterfour: MediaWiki Train: Creating new branch wmf/1.30.0-wmf.12 from master. See T168053 for deployment blockers.
  • 16:17 dcausse: restarting elastic on relforge100x servers to pick up new version of the plugins
  • 15:54 bblack: varnish backend restart on cp1072 (mailbox lag)
  • 15:43 bblack: rebooting lvs1002
  • 15:42 marostegui: db1069: Migrate trwiktionary.page from TokuDB to InnoDB
  • 15:40 bblack: stopping pybal on lvs1002 for impending reboot
  • 15:39 bblack: stopping pybal on 1002 for impending reboot
  • 15:33 marostegui: Stop s3 on db1069 - replication stuck
  • 15:17 marostegui: Stop MySQL on db1055 for maintenance - https://phabricator.wikimedia.org/T148507
  • 14:56 andrewbogott: rebooting labvirt1016
  • 14:46 marostegui: Deploy InnoDB compression on s3 - db2074 for the following tables (revision, pagelinks and templatelinks) - T170662
  • 14:45 ema: lvs1001-1003 (eqiad primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 14:28 ema: lvs1004-1006 (eqiad secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 14:27 bblack: restart varnish backend on cp1049 (mailbox lag)
  • 14:12 bblack: restart varnish backend on cp1074 (mailbox lag)
  • 13:53 ema@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,service=pdns_recursor
  • 13:26 ema@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,service=pdns_recursor
  • 13:10 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis except Commons (duration: 00m 44s)
  • 13:06 marostegui: Compress s2 on db1102 - T172169
  • 12:49 elukey: restart hive daemons on analytics1003 to pick up new jvm settings (bigger Xmx, JMX ports)
  • 12:06 dcausse: 100% cpu spike on elastic1023 caused percentiles to jump for a short period of time (T169498)
  • 12:04 elukey: stop eventlogging_sync on analytics-slaves && rename all CookieBlock* tables (log db) to CookieBlock*_backup - T171883
  • 11:52 marostegui: Stop MySQL on db2057 to copy its data to db2074 - T170662
  • 11:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2057 - T170662 (duration: 00m 43s)
  • 11:10 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T170662 (duration: 00m 43s)
  • 10:31 hashar: Enabling Zuul/CI again and reenabling puppet on contint1001
  • 10:24 hashar: contint1001 stopped puppet agent to prevent Zuul server to come back up
  • 10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
  • 08:55 ema: lvs2001-2003 (codfw primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 08:32 ema: lvs2004-2006 (codfw secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 08:03 ema: lvs3*: upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 07:40 ema: lvs4001, lvs4002 (ulsfo primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 07:35 ema: lvs4003, lvs4004 (ulsfo secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
  • 07:33 ema: pybal 1.13.11 uploaded to apt.w.o T103882
  • 06:20 marostegui: Stop MySQL on db2065 to copy its data to db2073 - T170662
  • 06:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 - T170662 (duration: 00m 43s)
  • 05:21 marostegui: Restart MySQL on labsdb1003 as it is totally stuck
  • 03:50 mobrovac@tin: Finished deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897 (duration: 07m 57s)
  • 03:42 mobrovac@tin: Started deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897
  • 02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 1 02:32:17 UTC 2017 (duration 6m 36s)
  • 02:25 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 46s)


Archives