You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Labslogbot
(krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting. (ori))
imported>Stashbot
(krinkle@deploy1002: Synchronized tests/: I60edfb0f60 (1/3) (duration: 03m 32s))
 
Line 1: Line 1:
== 2015-12-06 ==
== 2022-07-01 ==
* 21:48 ori: krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting.
* 01:39 krinkle@deploy1002: Synchronized tests/: {{Gerrit|I60edfb0f60}} (1/3) (duration: 03m 32s)
* 18:49 legoktm: reset auth token for User:QuimGil
* 01:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage
* 05:50 mutante: silver gzip /var/log/nutcracker.log.1
* 01:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 05:40 mutante: silver: apt-get clean for disk space
* 01:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 03:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Dec  6 03:57:02 UTC 2015 (duration 1h 31m 41s)
* 01:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 04s)
* 01:31 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage
* 01:31 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 01:30 krinkle@deploy1002: Synchronized src/: {{Gerrit|I796f38d0f04600c}} (3/3) (duration: 03m 24s)
* 01:26 krinkle@deploy1002: Synchronized multiversion/: {{Gerrit|I796f38d0f04600c}} (2/3) (duration: 03m 32s)
* 01:23 krinkle@deploy1002: Synchronized tests/: {{Gerrit|I796f38d0f04600c}} (1/3) (duration: 03m 41s)
* 01:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
* 01:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply
* 01:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply
* 01:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply
* 01:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2162.codfw.wmnet with OS bullseye
* 01:12 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS bullseye
* 01:02 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: host reimage
* 01:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS bullseye
* 00:57 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: host reimage
* 00:53 ejegg: updated payments-wiki from {{Gerrit|ef53c82e}} to {{Gerrit|78dee85e}}
* 00:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2168.mgmt.codfw.wmnet with reboot policy FORCED
* 00:51 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2167.mgmt.codfw.wmnet with reboot policy FORCED
* 00:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage
* 00:42 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage
* 00:37 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2162.codfw.wmnet with OS bullseye
* 00:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2168.mgmt.codfw.wmnet with reboot policy FORCED
* 00:28 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2167.mgmt.codfw.wmnet with reboot policy FORCED
* 00:27 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2166.mgmt.codfw.wmnet with reboot policy FORCED
* 00:26 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2165.mgmt.codfw.wmnet with reboot policy FORCED
* 00:23 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host db2161.codfw.wmnet with OS bullseye
* 00:05 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2166.mgmt.codfw.wmnet with reboot policy FORCED
* 00:05 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2163.mgmt.codfw.wmnet with reboot policy FORCED
* 00:01 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host db2165.mgmt.codfw.wmnet with reboot policy FORCED


== 2015-12-05 ==
==Archives==
* 18:30 gwicke: started nodetool decommission on restbase1008
See [[Server Admin Log/Archives]].
* 11:35 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s)
<noinclude>
* 11:23 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.5
[[Category:SAL]]
* 11:22 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s)
[[Category:Operations]]
* 11:13 logmsgbot: reedy@tin Synchronized docroot and w: Add jobqueue-labs to noc (duration: 00m 28s)
</noinclude>
* 08:59 bblack: offlined db1019 megacli disk 32:11
* 06:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Dec  5 06:09:07 UTC 2015 (duration 3h 44m 18s)
* 02:24 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s)
 
== 2015-12-04 ==
* 22:20 ejegg: updated dash from bed9895563fc0f5d3b96791fae6ed104b7778f0d to 9f0a4d98586f1281716e39b34d924de524458177
* 21:44 andrewbogott: disabling puppet on labcontrol1002 for ldap testing
* 21:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/Hooks.php: Iba0138a: Don't install a custom error handler for hooks (T117553) (duration: 00m 28s)
* 21:26 awight: Update fundraising CRM from 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a to f0ef0dab1fa7a85426a000c35750093d541fe4f5
* 20:28 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: Idee6a1980: job queue: use instances on port 6378 as aggregators (duration: 00m 30s)
* 19:21 ori: krypton: updated Grafana to 2.6.0-beta1 for bug fix for issue 3422
* 15:52 Jeff_Green: add mx record for donate.wikimedia.org
* 15:33 godog: ms-be2019 rebooted by itself, ilo event log shows "Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000038, Bank 0x00000003, Status 0xFE000040'00020135, Address 0x00000000'FEB82F63, Misc 0x00000000'00002285)"
* 08:52 godog: reimage restbase1009
* 05:59 gwicke: ran systemctl mask cassandra on restbase1009; it is important that this node does not start up.
* 05:53 gwicke: moved /var/lib/cassandra out of the way in an attempt to stop puppet restarting cassandra on decommissioned restbase1009
* 05:49 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Dec  4 05:49:46 UTC 2015 (duration 3h 21m 36s)
* 02:28 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 19s)
* 02:15 ori: CirrusSearch-common.php sync was for I826d000ca: Turn off backoff throttling of CirrusSearch jobs
* 02:15 logmsgbot: ori@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 29s)
* 01:33 bd808: Updated scholarships.wikimedia.org to af73bf6
* 00:35 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/CentralNotice: SWAT (duration: 00m 32s)
* 00:01 ejegg: pointed SmashPig IPN listener to live Adyen account
 
== 2015-12-03 ==
* 23:09 bblack: restarting pybal (w/ BGP enabled) on lvs100[123] (newly-installed w/ jessie)
* 22:59 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/includes/jobqueue/JobRunner.php: temporarily disable job throttling (duration: 00m 29s)
* 22:35 cwd: updated paymentswiki configuration [redux]
* 22:15 cwd: updated paymentswiki configuration
* 22:09 bd808: Removed zirconium.wikimedia.org from Trebuchet minions list for scholarships/scholarships
* 22:04 bd808: Updated scholarships.wikimedia.org to cb94319 plus local i18n filtering
* 21:48 Reedy: finished removing bogus msg_resource rows
* 21:28 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: re-sync (re-merged the change) (duration: 00m 29s)
* 21:27 bd808: Applied database migrations and purged last year's data from Wikimania Scholarships db
* 21:21 ottomata: restarted eventlogging with 4 mysql consumer processes running in parallel
* 21:21 bblack: rebooting lvs100[123] for reinstall to jessie
* 21:18 Reedy: Cleaning up msg_resource rows with bogus language codes
* 21:15 gwicke: stopped cassandra on 1009 as it's decommissioned & will be reimaged
* 21:13 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Re-fix the jobqueue on wikitech after redis cleanup (duration: 00m 26s)
* 20:55 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 47s)
* 20:45 _joe_: opening connection from mw1001 to silver, mysql
* 20:29 ori: on palladium: salt -G 'cluster:jobrunner' cmd.run 'service jobrunner status | grep running && service jobrunner restart'  ;  salt -G 'cluster:jobrunner' cmd.run 'service jobchron status | grep running && service jobchron restart'
* 20:28 ori: ran srem jobqueue:aggregator:s-wikis:v2 labswiki on rdb1001 aggr
* 19:41 bblack: disabling pybal on lvs100[123] over the next few minutes (for reinstall to jessie later after confirmation everything is still ok on [456])
* 19:10 jynus: restarting eventlogging_sync on db1047 and dbstore1002
* 19:04 jynus: starting m4 slave again on dbstore2002
* 18:45 andrewbogott: disabling puppet on labcontrol1002 to test openldap with pdns
* 18:33 mutante: neon - remove icinga user from "dialout" group
* 18:27 jynus: disabling eventlogging_sync process on dbstore1002 and db1047 and replication on the other m4 slaves
* 18:18 jynus: disabling event scheduler on db1046 (m4-master)
* 17:26 ejegg: updated fundraising dashboard from 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c to bed9895563fc0f5d3b96791fae6ed104b7778f0d
* 17:03 logmsgbot: kartik@tin Finished scap: Update ContentTranslation (duration: 05m 52s)
* 16:57 logmsgbot: kartik@tin Started scap: Update ContentTranslation
* 16:50 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: Fix the jobqueue on wikitech (duration: 00m 28s)
* 15:23 andrewbogott: stopping pdns on labcontrol2001
* 15:11 moritzm: restarting cassandra on restbase100[56] (subsequently) to effect openjdk security update
* 14:57 mobrovac: restbase end of deployment of 262da91a
* 14:48 mobrovac: restbase start deployment of 262da91a
* 14:06 moritzm: installed dpkg updates across the cluster
* 11:35 moritzm: restarting cassandra on aqs cluster (subsequently) to effect openjdk security update
* 10:51 jynus: restarting, upgrading and general maintenance for es1013 (depooled)
* 10:36 _joe_: imported dh-python into precise/universe from the ubuntu cloud archive
* 10:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1013 for maintenance (duration: 00m 30s)
* 05:50 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Dec  3 05:50:08 UTC 2015 (duration 50m 7s)
* 02:32 ejegg: updated CiviCRM from d5afa10383cfd41adbe798c3ed2aa97839be8bc2 to 8a3deedc04e4d3d1c1f78cb5b7ace7d8670f307a
* 02:25 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 09m 54s)
* 00:18 awight: Deploy paymentswiki config for T112181
 
== 2015-12-02 ==
* 23:47 ejegg: updated fundraising dashboard from 01a82b5e4909075e3190496e7fcda2b60f8d5d3c to 8412a2f5f91bb3a9f7873d6ac9fd5d7884eb988c
* 23:07 cwd: updated payments from 02a3771cae6e4e74d9d754a8b075de20aa171adc to ff7a706219e72f7362f4e06386e6732933a40478
* 22:09 jynus: unscheduled restart of dbstore1002 (analytics-slave)
* 21:44 jynus: disabling all alert notifications for dbstore1002
* 21:30 bblack: rebooting lvs1007 for interface config test (not active, no BGP)
* 20:37 Jeff_Green: enable mail queue monitoring for fundraising
* 19:13 jynus: restarting mysql on db2067 to test a configuration change
* 18:07 mutante: mw1136 - hhvm restart
* 14:45 Coren: deploying cleanup of labs PAM configuration - this should be a functional noop but may cause some puppet noise
* 12:41 akosiaris: restart cassandra on maps-test200{1,2,3,4}.codfw.wmnet
* 11:23 moritzm: restarting cassandra on restbase100[78] (subsequently) (to effect openjdk security updates plus related libs)
* 11:06 moritzm: restarting cassandra on restbase100[2-4] (subsequently) (to effect openjdk security updates plus related libs)
* 11:04 moritzm: restart cassandra on restbase1001 (to effect openjdk security updates plus related libs)
* 10:21 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 27s)
* 10:20 logmsgbot: aude@tin Synchronized dblists/arbitraryaccess.dblist: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 30s)
* 10:14 _joe_: clearing the job cache for ocg1003
* 09.43 +
* 09:22 _joe_: stopped ocg service on ocg1003
* 08:38 _joe_: planet1001 recovered as soon as I did get into console via gnt-instance
* 06:25 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.7/extensions/NavigationTiming: Idb675cdce: Add isHiDPI and isHttp2 properties; drop isHttps (T119014) (duration: 00m 50s)
* 05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Dec  2 05:53:21 UTC 2015 (duration 53m 20s)
* 02:28 mutante|away: labcontrol2001 - disable puppet, kill from puppet stored configs
* 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 47s)
* 00:29 mutante: puppetstoredconfigclean.rb labcontrol2001.wikimedia.org  fixes icinga config
 
== 2015-12-01 ==
* 22:41 mutante: stop/start ntp service on lvs2002
* 21:04 ejegg: updated fundraising from 2ee15180150c8b685b14abdbf7c6769736fb49a0 to 01a82b5e4909075e3190496e7fcda2b60f8d5d3c
* 20:54 mutante: re-enabled puppet on cache::misc, replaced planet cert
* 20:19 mutante: temp. disabling puppet on cache::misc
* 18:14 twentyafterfour: restarted apache on iridium to deploy https://phabricator.wikimedia.org/rPHEXd724c51a4144f7de548546875097da57ee1b38d7
* 17:05 moritzm: restarting cassandra instances on restbase200[3-6] (to effect openjdk security updates and a few depending libs)
* 17:00 moritzm: restarting cassandra instances on restbase2002 (to effect openjdk security updates and a few depending libs)
* 16:55 moritzm: restarting cassandra instances on restbase2001 (to effect openjdk security updates and a few depending libs)
* 16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all accounts on eswiki. Part II [[gerrit:250474]] (duration: 00m 28s)
* 16:21 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor for all accounts on eswiki. Part I [[gerrit:250474]] (duration: 01m 29s)
* 15:30 ejegg: changed donation qc time limit from 90 to 100 sec
* 12:59 godog: reimage mw2208
* 10:50 moritzm: restarting on cassandra instances on restbase1008 (to effect openjdk security updates and a few depending libs)
* 06:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Dec  1 06:52:40 UTC 2015 (duration 52m 39s)
* 02:26 logmsgbot: mwdeploy@tin sync-l10n completed (1.27.0-wmf.7) (duration: 10m 14s)
* 02:18 mutante: restarted hhvm on mw1147
* 01:47 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 30s)
* 00:51 ejegg: updated fundraising dash from bc0ec4bb2427ebeb2d4cd60eda65e3d82d006937 to 2ee15180150c8b685b14abdbf7c6769736fb49a0
* 00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor: SWAT (duration: 00m 29s)
* 00:47 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: SWAT (duration: 00m 30s)
* 00:45 ejegg: updated Adyen credentials in SmashPig IPN listener
* 00:39 ejegg: updated Adyen credentials on payments-wiki
* 00:35 ejegg: updated payments-wikifrom 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 02a3771cae6e4e74d9d754a8b075de20aa171adc
 
== 2015-11-30 ==
* 22:53 ejegg: updated fundraising dash from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
* 22:37 ejegg: updated fundraising dashboard from 5a6b2dda71e6ce76d7bbba853acae8dc9416052c to 5afc7b02afbd9385db80222271d3d8cc64c1ea13
* 22:16 awight: Configuring paymentswiki to not exlicitly include DefaultSettings.php
* 21:55 mutante: re-wrote l10nupdate cron; restarted cron service on tin
* 21:53 awight: Tweak paymentswiki-staging config: remove line that exlicitly includes DefaultSettings.php
* 20:05 apergos: re-enabled puppet on neodymium, minion testing concluded for now
* 19:47 gwicke: running `nodetool decommission` on restbase1009 in preparation for the conversion to the multi-instance setup, per https://phabricator.wikimedia.org/T95253#
* 19:31 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm deprecated/unused rate limit log config (duration: 00m 28s)
* 17:27 logmsgbot: demon@tin Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/: need maint script errywhere (duration: 00m 28s)
* 16:51 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/extensions/ContentTranslation/modules/draft/ext.cx.draft.js: SWAT: Add some extra information to save failure logging [[gerrit:255956]] (duration: 00m 28s)
* 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable QuickSurveys reader segmentation survey [[gerrit:255448]] (duration: 00m 28s)
* 16:30 paravoid: mw1002 service hhvm restart
* 16:17 paravoid: rolling back to kernel 3.19 on lvs2001/2/3
* 15:29 paravoid: stopping pybal on lvs2001/2/3
* 15:21 paravoid: switching lvs2004/5/6 traffic back to lvs2001/2/3
* 15:13 paravoid: switching lvs2001/2/3 traffic to lvs2004/5/6 and upgrading kernels
* 15:12 _joe_: restarting HHVM on mw1147 too, same reason as mw1114
* 15:10 _joe_: restarting hhvm on mw1114, stuck in __pthread_cond_wait () [folly::EventBase::runInEventBaseThreadAndWait ()], apparently blocked in writing to stdout
* 15:02 paravoid: switching traffic from lvs4002 to lvs4004; upgrading lvs4002's kernel
* 15:02 paravoid: switching traffic back to lvs4001
* 14:57 paravoid: switching traffic from lvs4001 to lvs4003; upgrading lvs4001's kernel
* 14:45 paravoid: switching traffic from lvs3001 to lvs3003; upgrading lvs3001's kernel
* 14:38 paravoid: switching traffic back to lvs3002
* 14:31 paravoid: switching traffic from lvs3002 to lvs3004; upgrading lvs3002's kernel
* 14:07 bblack: upgrading varnishkafka package on all caches
* 13:52 bblack: updating varnishkafka on cp1065
* 11:03 godog: upgrade python-statsd to 3.0.1 in eqiad
* 10:59 godog: upgrade python-statsd to 3.0.1 in codfw
* 10:15 godog: reenable puppet on graphite1001
* 10:10 paravoid: re-enabling OSPF over cr2-eqiad:xe-5/2/2 <-> cr1-ulsfo:xe-0/0/3.538
* 10:09 paravoid: re-enabling cr2-eqiad:xe-5/2/0 and xe-5/2/1
* 10:01 jynus: performing schema change on db1046 (analytics master)
* 09:32 jynus: removing old snapshots from db1046
* 06:39 ori: Restarted statsv on hafnium
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
* 01:56 gwicke: started `nodetool cleanup` on restbase1002 to get rid of unnecessary data from earlier 1001 decommission attempt
* 01:05 logmsgbot: bd808@tin sync-l10n completed (1.27.0-wmf.7) (duration: 01m 19s)
* 01:04 bd808: testing l10n cache rebuild as l10nupdate user (take 2)
* 00:57 Krenair: test
 
== 2015-11-29 ==
* 21:25 jynus: importing user.user_touched (s7) from dbstore1002 to sanitarium. s7 lag on labs replicas will be higher for some minutes.
* 20:51 jynus: importing user.user_touched (s6) from dbstore1002 to sanitarium. s6 lag on labs replicas will be higher for some minutes.
* 20:28 jynus: importing user.user_touched (s5) from dbstore1002 to sanitarium. s5 lag on labs replicas will be higher for some minutes.
* 19:51 jynus: importing user.user_touched (s4) from dbstore1002 to sanitarium. s4 lab will be affected for some minutes.
* 04:50 gwicke: restarted cassandra on restbase1009 to avoid it running out of disk space; had large compaction (~2TB) at 80% and only 64G disk space left
* 03:01 YuviPanda: run chown -R l10nupdate: /var/lib/l10nupdate/mediawiki  for Reedy on tin
* 02:28 Reedy: l10nupdate failed because some git objects owned by 997:l10nupdate
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
 
== 2015-11-28 ==
* 22:48 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/cache/l10n: bd808 testing l10nupdate sync-dir using stale branch (duration: 01m 29s)
* 20:49 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
* 20:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 07m 11s)
* 20:35 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Ie33ae3b6a: Increase $wgCopyUploadTimeout to 90 seconds (from default 25) (T118887) (duration: 00m 27s)
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
 
== 2015-11-27 ==
* 21:23 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/255726/1 (duration: 00m 31s)
* 17:46 paravoid: re-enabling cr2-eqiad:xe-5/2/2 (link to cr1-ulsfo) and xe-5/2/3 (link to cr2-codfw)
* 15:08 godog: kill stray cpu hog xelatex on ocg1003, orphan and started on oct22
* 14:16 Jeff_Green: fundraising database maintenance
* 12:36 godog: ban some grafana dashboards from graphite
* 10:56 apergos: restarted gitblit on antimony about 40 minutes ago, it takes about 20 minutes to load up caches of commits, now responding but... it's taking 17-18 seconds to load repos instead of 1-3 as normal. so timeouts
* 10:47 jynus: importing user.user_touched (s3) from dbstore1002 to sanitarium, it will affect s3 labsdbs lag
* 10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repooling previouly problematic servers (duration: 00m 28s)
* 10:10 apergos: removed gitblit.log.1 on antimony, it was 58GB
* 10:03 mark: Reenabling cr2-eqiad:xe-5/3/0
* 09:53 mark: Disabled cr2-eqiad:xe-5/2/*
* 09:53 paravoid: issue has been identified as a a problematic PIC (5/2) on cr2-eqiad and is being worked around
* 09:51 mark: Set VRRP priority 50 on all cr2-eqiad VRRP sessions, making it backup
* 09:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling problematic servers (duration: 00m 27s)
* 09:48 mark: Disabling cr2-eqiad:xe-5/2/1
* 09:45 mark: Disabling cr2-eqiad:xe-5/{2,3}/0
* 09:42 paravoid: network troubles within eqiad still being investigated
* 09:42 paravoid: network troubles; ulsfo was depooled ~20 minutes ago due to backhauling issues
* 08:31 _joe_: rolling restart of all redis instances on rdb1001 and 1003 (later 1002/4) to fix the upstart bug
* 07:55 _joe_: restarting redis rdb1003:6380 to test rdb disabling
* 06:57 ori: ran eventloggingctl stop / eventloggingctl start on eventlog1001
* 06:18 ori: coal metrics flat since 01:25:00 AM UTC
* 02:47 Reedy: manually running dsh -g mediawiki-installation -M -F 40 -- "sudo -u mwdeploy /srv/deployment/scap/scap/bin/scap-rebuild-cdbs"
* 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate failed: Failed to sync-dir 'php-1.27.0-wmf.7/cache/l10n'
* 02:45 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 09m 10s)
* 02:14 Reedy: killed pre 1.27 l10n cache files from tin:/var/lib/l10nupdate/caches
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
 
== 2015-11-26 ==
* 20:59 logmsgbot: aaron@tin Synchronized wmf-config/jobqueue-eqiad.php: Adjust connection timeout and maxPartitionsTry (duration: 00m 34s)
* 16:50 jynus: backfilling user.user_touched from dbstore1002 to sanitarium on all wikis
* 16:30 paravoid: switch lvs3001 traffic to lvs3003dd
* 16:22 paravoid: rebooting lvs3003 with Linux 4.3-1~wmf1
* 15:58 Krinkle: Deployed patch for T119707
* 15:49 godog: reenable puppet on graphite1001
* 14:30 _joe_: run /usr/local/bin/foreachwiki updateSpecialPages.php manually in a screen on terbium
* 13:27 moritzm: restarted opendj on nembus for export of LDAP data
* 13:25 moritzm: stopping opendj on nembus for export of LDAP data
* 11:54 apergos: disabling puppet on neodymium for salt minon tracing
* 11:35 apergos: re-enabled puppet on neodymium, config testing done for now
* 11:32 godog: drop local_group_globaldomain_T_mathoid_data and local_group_globaldomain_T_mathoid_request CF stats from graphite
* 11:25 jynus: manually executing some updateSpecial pages script to try to debug its problems
* 11:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1053 after maintenance (duration: 00m 28s)
* 11:13 apergos: disabled puppet on neodymium for salt master testing
* 09:49 moritzm: restarted cassandra nodes on the restbase staging cluster to pick up openjdk security updates
* 09:38 jynus_: upgrading/rebooting/restarting mysql on db1053 (depooled)
* 09:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1053 for maintenance (duration: 00m 27s)
* 07:15 ori: rdb100x migrated to redis::instance, meaning they each have two additional redis instances, on port 6380 and 6381, accordingly.
* 06:46 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I3e6021ed3: Update job queue config to use 2 new rdb1003 instances (duration: 00m 28s)
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
* 01:31 awight: updating paymentswiki-staging from 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe to 988ba71418c370e0365a102a0b87f823671ea1e3
* 01:03 awight: from f867beecd3a4fb596e2d08c8d2b0e4173ad0b356 to d5afa10383cfd41adbe798c3ed2aa97839be8bc2
* 00:14 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I1e0d0ed8215f: pool rdb1007:6381 (duration: 00m 30s)
 
== 2015-11-25 ==
* 23:21 logmsgbot: krenair@tin Synchronized private/README_BEFORE_MODIFYING_ANYTHING: 334ca105e92aaf7046e244ff39189f3823d31a7d (duration: 00m 32s)
* 22:14 logmsgbot: demon@tin Finished scap: new MW release, swapping extdist config + msgs (duration: 23m 59s)
* 21:50 logmsgbot: demon@tin Started scap: new MW release, swapping extdist config + msgs
* 20:31 gwicke: running `nodetool cleanup` on restbase1007 to make sure that we don't have extra sstables from the 1001 decommision taking up space
* 19:13 mobrovac: restbase deploy end of 74662c
* 19:13 moritzm: installed django security updates on stat* and graphite hosts
* 18:47 robh: removing cr2-ulsfo:xe-1/2/0, Patch ID 1062 as T118171 cancels that link
* 18:46 mobrovac: restbase deploy start of 74662c
* 18:19 logmsgbot: bd808@tin Synchronized README: Testing l10nupdate uid fix for T119165 (duration: 00m 28s)
* 18:04 andrewbogott: restored pdns-recursor on holmium, again
* 18:00 mobrovac: restbase canary deploy to restbase1001 of 74662c6
* 17:44 andrewbogott: killing pdns-recursor on holmium
* 16:53 andrewbogott: killing pdns-recursor on holmium
* 16:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 47s)
* 15:53 godog: ban grafana kafka dashboard temporarily from graphite
* 14:57 godog: bounce uwsgi on graphite1001
* 14:52 godog: stop puppet on restbase1* / restbase2* before https://gerrit.wikimedia.org/r/#/c/254372/
* 13:09 logmsgbot: hashar@tin Synchronized php-1.27.0-wmf.7/Rakefile: Added Rakefile https://gerrit.wikimedia.org/r/#/c/254423/ (duration: 00m 28s)
* 11:19 jynus: applying ferm and p_s to db1044 (depooled)
* 10:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1044 for a brief maintenance (duration: 00m 28s)
* 08:08 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9bea66df: Update job queue config to use 2 rdb1007 instances (duration: 00m 28s)
* 08:06 akosiaris: update puppet compilers fact database
* 03:44 Tim: on palladium ran puppet post-merge hook manually after puppet-merge failed with "error: Ref refs/remotes/origin/production is at 380706c5bd2f47d140ffa183dd22fe920927af7e but expected 897486120d645f1c30dcf68bfb3198e2f39ec639"
* 03:35 logmsgbot: krinkle@tin Synchronized portals/prod/wikipedia.org/index.html: Fix Uncaught TypeError: innerHTML() is not a function (duration: 00m 28s)
* 03:12 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I4e21eda0f3 (duration: 01m 06s)
* 02:40 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 06m 50s)
* 02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of core failed
* 01:31 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.7/StartProfiler.php: create symlink (duration: 01m 10s)
* 01:03 mutante: re-armed keyholders on mira
* 00:46 mutante: starting redis on rdb1007 (it died and was status: dead)
* 00:40 mutante: tin: fixing l10nupdate UID (997->10002), file ownership
* 00:37 ori: Migrating rdb1007 from redis::legacy to redis::instance; will involve a service restart.
* 00:21 ejegg: updated payments-wiki from 5358960d6857f589939c9b86ea37af168a4f8b0b to 7fb9d6599c3833f0a55500ab77abd2b7ffe74cfe
* 00:20 mutante: rebooting mira
* 00:14 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255276/ (duration: 00m 28s)
* 00:13 awight: Disabled Amazon on donatewiki
* 00:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/254443/ (duration: 00m 28s)
 
== 2015-11-24 ==
* 23:43 mutante: fixinx l10nupdate UID / file ownership on mira
* 23:40 csteipp: deployed patch for T119309
* 23:36 ostriches: phabricator: restarted daemons, were complaining about out of sync config
* 22:07 ejegg: updated SmashPig from 6243d4f172dbfa5e3820f2e1e3a3165efff58bac to b3ee85aab38fdeef3f045ba1733f582e9a5006b2
* 21:39 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: I80d38773057 (duration: 00m 28s)
* 20:31 mutante: gnt-instance reboot seaborgium.wikimedia.org
* 20:01 bblack: disabling puppet on neon to avoid monitoring race-condition spam
* 19:30 moritzm: restarted hadoop workers on analytics1028 to analytics1055 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
* 19:27 bblack: misc-cluster: all 4x nodes on new config and pooled
* 19:18 bblack: misc-cluster: cp1056+7 pooled (new config), cp1069+70 depooled (old config)
* 18:52 bblack: repooled cp1056, depooled cp1057 (misc 2layer)
* 18:38 YuviPanda: restarted apache2 on etherpad
* 18:30 bblack: depooling cp1056 for misc-cluster 2layer work...
* 17:59 godog: nodetool cleanup on restbase1007
* 17:20 godog: stop decommissioning restbase1001, bounce cassandra
* 16:40 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/resources/lib/oojs-ui/oojs-ui.js: SWAT: OOjs UI: Backport 4fbbc737c86b500c11bbb471ec1001c50ab8853c [[gerrit:255032]] (duration: 00m 28s)
* 16:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable es labs replica writes for nlwiki, frwiki and eswiki [[gerrit:255122]] (duration: 00m 27s)
* 16:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Fix article-recommender-1 campaign [[gerrit:255069]] (duration: 00m 27s)
* 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Third QuickSurveys external survey [[gerrit:254908]] (duration: 00m 30s)
* 16:08 paravoid: starting pybal on lvs3001 again, testing is over
* 16:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all new accounts on eswiki [[gerrit:250473]] (duration: 00m 46s)
* 15:38 moritzm: restarted analytics1030 to pick up openjdk security update (plus updates for libpng, nss, nspr, pixbuf and libxml as used by openjdk)
* 15:33 hashar: enabled puppet on both gallium and scandium
* 15:27 hashar: disabling puppet on both gallium and scandium
* 14:24 godog: upgrade diamond to 3.5-4 on precise hosts in eqiad
* 14:22 godog: upgrade diamond to 3.5-4 on precise hosts in ulsfo
* 13:56 godog: upgrade diamond to 3.5-4 on precise hosts in esams
* 12:18 paravoid: stopping pybal on lvs3001, switching its traffic to lvs3003
* 11:48 paravoid: rebooting lvs3003 with linux 4.2.6-1~bpo8+wmf1
* 10:28 jynus: dropping user_daily_contribs view from labsdb hosts
* 10:07 godog: stop ircecho on neon temporarily while https://gerrit.wikimedia.org/r/#/c/255094/ applies
* 08:24 jynus: dbstore1002 (analytics-slave) mysql is not responding, forcing restart
* 05:58 _joe_: powercycling ms-be1010, xfs kernel lockups
* 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 34s)
* 02:12 ejegg: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to 6243d4f172dbfa5e3820f2e1e3a3165efff58bac
* 01:00 ejegg: updated payments-wiki from ecbc49c63b259a4363c51491ddff273e3579f0e9 to 5358960d6857f589939c9b86ea37af168a4f8b0b
* 00:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.7/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/255053/ (duration: 00m 28s)
* 00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable commonswiki for elasticsearch labs replica writes (duration: 00m 28s)
* 00:15 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable wikidatawiki for elasticsearch labs replica (duration: 00m 27s)
 
== 2015-11-23 ==
* 23:55 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 28s)
* 23:30 awight: campaigns reenabled
* 23:29 awight: reenabling paymentswiki
* 23:28 awight: reenabling fundraising-jenkins jobs
* 23:08 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable elasticsearch labs replica for enwiki and dewiki (duration: 00m 28s)
* 23:04 awight: shutting down Jenkins
* 22:52 awight: disable paymentswiki in config
* 21:16 hoo: Updated Wikidata's property suggester with data from today's json dump
* 21:15 jynus: enforcing limits on running queries on all labsdb databases
* 20:47 jynus: restarting labsdb1002, hopfully for the last time
* 17:54 jynus: temporarily stopping labsdb1002 to solve replication issues
* 17:32 Coren: emergency hotfix on labstore1001 (apt-get remove nfsd-ldap) as it appears to not work in all situations.
* 17:30 ori: restarting hhvm on mw1122 after lock-up; backtrace saved as /tmp/hhvm.32016.bt.
* 17:28 logmsgbot: thcipriani@tin Finished scap: SWAT: Update CentralNotice [[gerrit:254865]] (duration: 26m 31s)
* 17:02 logmsgbot: thcipriani@tin Started scap: SWAT: Update CentralNotice [[gerrit:254865]]
* 16:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: noindex user namespace on en.wikipedia.org [[gerrit:237330]] (duration: 00m 28s)
* 16:43 jynus: restarting mysqld on labsdb1003 again
* 16:38 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: End first mobile QuickSurveys campaign [[gerrit:254831]] (duration: 00m 28s)
* 16:32 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.7/includes/specials/SpecialWatchlist.php: SWAT: Special:Watchlist: Add user preference to "Show last" options, fix float comparison [[gerrit:254880]] (duration: 00m 29s)
* 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable article-recommender-1 campaign as default [[gerrit:254117]] (duration: 00m 52s)
* 15:56 paravoid: rebooted lvs3003 back with (an updated) 3.19
* 15:53 _joe_: live-resizing /var/lib/wdqs on wdqs1002
* 15:15 Coren: NFS server on labstore1001 restarted with no issues.
* 15:11 Coren: Restarting NFS daemon on labstore1001 w/ the new LDAP shim.
* 14:53 paravoid: powercycling cp1053, unresponsive, blank console
* 14:53 bblack: depooling cp1053
* 14:50 ottomata: removing erbium udp2log varnishncsa instances from all caches
* 14:38 paravoid: rebooting lvs3003 with new kernel
* 13:18 jynus: restarting mysql on labsdb1003, unresponsive
* 11:05 _joe_: stopping apache on mw2051 for testing
* 08:58 godog: restbase1001 nodetool decommission
* 08:57 mobrovac: mathoid deploying ff03d56
* 08:12 moritzm: restarted gitblit on antimony
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 57s)
* 01:27 Jamesofur: end enwiki test election from cli to avoid confusion with real election
 
== 2015-11-22 ==
* 21:30 Reedy: restarting stuck Jenkins
* 02:16 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 22s)
 
== 2015-11-21 ==
* 19:14 ejegg|away: updated fundraising tools from d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132 to ebed29c0eccf38c812b20a957b3487a15bfa9cbc
* 18:43 ejegg|away: rolled back SmashPig d7534bde8c247a1e877a4bcd768b9963f89666d7 --> 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
* 18:35 ejegg|away: updated SmashPig from 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e to d7534bde8c247a1e877a4bcd768b9963f89666d7
* 03:19 ori: SecurePoll arbcomlist.php run for enwiki (requested by Jamesofur) complete.
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 05m 18s)
* 00:50 ori: update-ubuntu-mirror on carbon completed successfully ; going to re-enable icinga-wm
* 00:46 ori: rampant puppet failures appear to be the result of a Hash Sum mismatch error during apt-get update; attempting to fix by running sudo -u mirror /usr/local/sbin/update-ubuntu-mirror on carbon.
* 00:37 ori: disabled puppet on neon for one hour and killed icinga-wm to silence a tide of puppet failure alerts
* 00:24 logmsgbot: demon@tin Synchronized .arcconfig: no-op, for completeness (duration: 00m 31s)
 
== 2015-11-20 ==
* 21:45 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.7/includes/User.php: Deploy fix for T119021 (duration: 00m 28s)
* 18:55 ori: resuming interrupted arbcomlist invocation on terbium
* 16:51 _joe_: decommissioning mw1041: removed facts and certs, removed from conftool and pybal
* 16:44 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Add throttle config for Viquimarató nit digital/2015 (duration: 00m 28s)
* 16:17 _joe_: depooled mw1041
* 15:41 Reedy: running sync-common on mw1041
* 15:32 _joe_: powercycling mw1041
* 15:26 logmsgbot: reedy@tin Synchronized wmf-config/throttle.php: Update throttle for event tomorrow (duration: 00m 28s)
* 14:35 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 3000
* 10:06 ori: Running SecurePoll's arbcomlist.php on terbium with performance fix from https://gerrit.wikimedia.org/r/#/c/254375/ , per request from Jamesofur. Looks to be behaving well, can be killed if it acts up.
* 07:44 _joe_: rebooted pybal-test2001
* 07:36 _joe_: powercycled mw1041
* 04:55 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: inconsequential sync to test mirroring (duration: 02m 14s)
* 04:41 logmsgbot: twentyafterfour@tin Synchronized multiversion/checkoutMediaWiki.php: deploying I2ec91aa33e05c2dc367db8a6f6ba56be5397906b to test scap mirroring (duration: 00m 20s)
* 04:36 twentyafterfour: deploying scap on tin
* 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 10m 47s)
* 01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/src/mediawiki.widgets/mw.widgets.TitleWidget.js: Fix conflicting configuration name in TitleInputWidget (duration: 00m 19s)
* 01:31 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/includes/widget/TitleInputWidget.php: Fix conflicting configuration name in TitleInputWidget (duration: 00m 20s)
* 01:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/resources/lib/jquery.i18n/src/jquery.i18n.language.js: Unbreak IE8 support in jquery.i18n (duration: 00m 25s)
* 01:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Echo/includes/formatters/BasicFormatter.php: unstub wgLang to fix type hint errors (duration: 00m 20s)
* 01:11 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Reduce sampling rate for QuickSurveys survey (28c69d2) (duration: 00m 20s)
* 00:37 awight: Updating fundraising/crm from e486ba424092866cdd8a15d26aeb17a412cd631a to f867beecd3a4fb596e2d08c8d2b0e4173ad0b356
* 00:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
* 00:17 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/253933 and https://gerrit.wikimedia.org/r/254070 (duration: 00m 19s)
* 00:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.7/extensions/CirrusSearch/: SWAT https://gerrit.wikimedia.org/r/254320 and https://gerrit.wikimedia.org/r/254319 (duration: 00m 20s)
* 00:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/254322/ (duration: 00m 20s)
 
== 2015-11-19 ==
* 23:35 awight: Update fundraising/tools from c6315eddf6f495e45401844d5f61bb240464c1e5 to d7253eb1c6c9478ce2cf8760279d2ad1b7f0a132
* 22:31 csteipp: deployed followup for T109724
* 22:21 csteipp: deployed patch for T118682
* 22:13 csteipp: deployed patch for T118032
* 22:11 csteipp: deployed patch for T97897
* 21:31 mutante: magnesium - apt-get upgrade
* 21:13 mutante: racktables moved to krypton and upgraded to 0.20.6
* 20:28 mutante: racktables - in migration - brb
* 20:22 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.7/extensions/QuickSurveys/: Deploy https://gerrit.wikimedia.org/r/#/c/254184/ (duration: 00m 19s)
* 20:19 awight: push paymentswiki config change, T115746
* 20:13 bblack: upgrading pybal to 1.13.1 on lvs400[12] and failing back traffic
* 19:51 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 22s)
* 19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.7
* 18:53 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.7/extensions/Wikidata: Adjust watchlist filter for core change (duration: 00m 30s)
* 16:24 godog: repool restbase1002
* 16:19 cmjohnson1: replacing pem 0 cr1-eqiad
* 14:55 godog: roll restart restbase in eqiad
* 14:41 godog: roll-restart restbase on aqs100*
* 14:40 moritzm: uploaded linux-meta 1.3 to carbon
* 14:39 moritzm: uploaded linux 3.19.3-9 to carbon
* 14:14 godog: roll-restart restbase in codfw
* 12:39 bblack: failing over ulsfo traffic to lvs400[34] (for 1.13.1 testing)
* 11:23 hashar: restarted zuul-merger on gallium. We now have two instances running (gallium and scandium)
* 11:21 hashar: stopped zuul-merger on gallium, started the one on scandium
* 11:00 jynus: modifying filters on all labs db replicas (sanitarium), small amounts of lag could be created
* 10:42 godog: hhvm-dump-debug --full and bounce hhvm on mw1147
* 10:40 godog: hhvm-dump-debug --full and bounce hhvm on mw1131
* 10:13 godog: disable puppet on deployment_target:restbase/deploy
* 10:08 hashar: going to restart Jenkins a few times
* 09:53 godog: delete old mobileapps CF stats from graphite1001 / graphite2001
* 06:19 YuviPanda: remove role::labs::instance puppetClass from all labs instances
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 58s)
* 01:31 awight: updating fundraising/crm from 67b855cbce358d215b8025da6fe7031bf3d221d9 to e486ba424092866cdd8a15d26aeb17a412cd631a
* 01:30 twentyafterfour: finished phabricator upgrade
* 00:29 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.7/extensions/Flow/: SWAT: better permissions errors for opt-in (duration: 00m 21s)
* 00:19 RoanKattouw: Ran namespaceDupes.php on eswiktionary
* 00:13 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: T118844, T119006 (duration: 00m 19s)
* 00:12 logmsgbot: catrope@tin Synchronized portals: SWAT (duration: 00m 19s)
 
== 2015-11-18 ==
* 23:51 yurik: git deploy sync latest graphoid
* 23:49 bblack: upgrading pybal to 1.13.1 on lvs300[34]
* 23:45 bblack: upgrading pybal to 1.13.1 on lvs400[34]
* 22:44 awight: Updating fundraising/crm from 230cebb4a3b46a75fef758d46f4db22de29286cb to 67b855cbce358d215b8025da6fe7031bf3d221d9
* 22:38 mutante: powercycled analytics1030 - no ssh response, mgmt console login timed out
* 21:15 subbu: finished deploying parsoid sha e0a4fc91
* 21:10 subbu: synced code; restarted parsoid on wtp1003 as a canary
* 21:06 subbu: starting parsoid deploy
* 20:43 bblack: upgrading pybal to 1.13.1 on lvs200[456]
* 20:09 logmsgbot: maxsem reset User:人神之间 email (confirmed in person with Matanya)
* 19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.7
* 18:18 andrewbogott: repooled mw1160.  And that’s all of them!
* 18:09 andrewbogott: repooled mw1159, depooled mw1160
* 17:59 andrewbogott: repooling mw1157, depooled mw1159
* 17:48 andrewbogott: repooled mw1155, depooled 1157 (I did 56 and 58 list night)
* 17:36 mutante: releases.wikimedia.org now enforcing https protocol
* 17:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/includes/db/loadbalancer/LoadBalancer.php: I03e1386b: Make getLaggedSlaveMode() use reuseConnection() as needed (T118162) (duration: 00m 19s)
* 17:26 andrewbogott: repooled mw1154, depooled mw1155
* 17:17 andrewbogott: repooled mw1153, depooling mw1154
* 17:09 andrewbogott: depooled mw1153 for https://phabricator.wikimedia.org/T118888
* 17:07 thcipriani: updateCollation.php for srwiki 1670608 rows processed
* 17:05 godog: restart restbase2002-a cassandra instance, bootstrap failed with stream error
* 16:23 bblack: upgrading pybal on lvs1006 (1.13.1)
* 16:23 _joe_: upgrading pybal on lvs1006 (1.13.1)
* 16:21 bblack: upgrading pybal on lvs1005 (1.13.1)
* 16:21 bblack: upgrading pybal on lvs1004 (1.13.1)
* 16:16 thcipriani: updateCollation.php for srwiktionary complete 82713 rows processed
* 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation to "uca-sr" at srwiki and srwiktionary [[gerrit:253903]] (duration: 00m 20s)
* 16:11 bblack: upgrading pybal on lvs1004 (1.13.1)
* 16:07 mobrovac: restbase cassandra dropping old mobileapps keyspaces
* 16:05 mobrovac: restbase end deploy of 6b5a602
* 15:56 mobrovac: restbase start deploy of 6b5a602
* 15:50 bblack: upgrading pybal to 1.13 on lvs100[56] (with service clean)
* 15:37 jynus: warming up srwiki.page and srwiki.categorylinks on s3 slaves
* 15:29 mobrovac: restbase canary deploy to rb1001 of 6b5a602
* 15:28 bblack: upgrading pybal to 1.13 on lvs1004, with cleared tables
* 15:23 _joe_: uploaded pybal 1.13 to jessie-wikimedia
* 15:14 hashar: stopping zuul-merger on scandium. Lacks ssh private key to reach gerrit
* 14:58 hashar: Pooled zuul-merger instance on scandium.eqiad.wmnet . In case it screw up one has to stop the zuul-merger service on scandium.
* 14:57 godog: bounce mobileapps on scb1001/scb1002
* 14:47 mobrovac: mobileapps deploying 151c312
* 13:55 bblack: disabling pybal on lvs100[123] - fallback to lvs100[456]
* 12:00 hashar: reenabled puppet on gallium. Havent pooled zuul-merger yet
* 11:57 hashar: disabled puppet on gallium
* 11:56 hashar: pooling new zuul-merger on scandium
* 11:44 mark: Set OSPF/OSPFv3 metric 320 (instead of 360) on new Zayo link between cr2-eqiad:xe-5/2/3 and cr2-codfw:xe-5/0/1
* 10:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Decreasing weight of db1035 due to connection exhaustion (duration: 00m 19s)
* 10:31 hashar: gallium: apt-get upgrade
* 10:30 hashar: upgrading Zuul on gallium (CI interruption for a minute)
* 09:48 jynus: several db maintenance tasks on db1044 -optimization, upgrade- expect lag (node is depooled)
* 09:25 godog: reimage restbase2002
* 09:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1027, depool db1044 for maintenance (duration: 01m 08s)
* 09:09 _joe_: restarted HHVM on mw1095, stuck in treadmill::startrequest() on a contended lock
* 06:31 _joe_: rsyncing terbium homes from rutherfordium
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 10m 10s)
* 01:46 andrewbogott: repooling mw1156
* 01:43 yurik_: synced graphoid service
* 01:40 andrewbogott: updated kernel on mw1156, now rebooting.
* 01:35 andrewbogott: repooling mw1158, depooling mw1156
* 01:22 andrewbogott: rebooting mw1158
* 01:19 andrewbogott: upgrading mw1158 to 3.13.0-62
* 00:26 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: wgRCWatchCategoryMembership true on group0 wikis (duration: 00m 27s)
* 00:04 logmsgbot: maxsem@tin Synchronized portals/: SWAT (duration: 00m 32s)
 
== 2015-11-17 ==
* 23:43 mutante: deactivated wekipedia.com
* 23:40 YuviPanda: remove realm variable from all instances in ldap
* 23:18 mutante: deactivated wikiartpedia.[biz|co|info|me|mobi|net|org]
* 23:17 awight: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to ecbc49c63b259a4363c51491ddff273e3579f0e9
* 23:13 mutante: deactivated border-wikipedia.de
* 23:01 mutante: deactivated wikimedia.xyz
* 22:57 mutante: deactivated visualwikipedia.[com|net]
* 22:56 mutante: deactivated softwarewikipedia.[com|net|org]
* 22:54 mutante: deactivated indiawikipedia.com
* 22:52 mutante: deactivated vikipedi.com.tr, vikipedia.com.tr
* 22:38 awight: updated fundraising/tools from 182f217ea20a496fe0f9737447d04ed9a3bccde8 to c6315eddf6f495e45401844d5f61bb240464c1e5
* 22:23 mutante: deactivated wikimaps.com, wikimaps.net, wikimaps.org
* 22:20 mutante: deactivated wikimedia.biz, webhostingwikipedia.com
* 22:18 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.7
* 21:07 logmsgbot: twentyafterfour@tin Finished scap: sync new branch 1.27.0-wmf.7 and enable for testwiki (duration: 29m 40s)
* 20:39 mutante: deleted sitemap.wikimedia.org (T101486)
* 20:38 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
* 20:31 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command 'cp -r "/tmp/scap_l10n_2994655917"/* "/srv/mediawiki-staging/php-1.27.0-wmf.7/cache/l10n"' returned non-zero exit status 1 (duration: 00m 42s)
* 20:30 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
* 20:26 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4014066321" --threads=4 --lang en  --quiet' returned non-zero exit status 1 (duration: 01m 30s)
* 20:25 logmsgbot: twentyafterfour@tin Started scap: sync new branch 1.27.0-wmf.7 and enable for testwiki
* 18:54 andrewbogott: depooled mw1158 due to kernel errors.  https://phabricator.wikimedia.org/T118888
* 18:10 cmjohnson1: swapping failing disk 32:8 on db1027..will cause icinga alert (jynus)
* 18:02 bblack: upgrading and re-enabling pybal on lvs300[12].esams.wmnet (1.10 -> 1.12)
* 17:54 yurik: deployed latest graphoid service
* 16:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation for bs.wikipedia.org [[gerrit:250862]] (duration: 00m 26s)
* 16:42 bblack: upgrading / re-enabling pybal on lvs200[123].codfw.wmnet (1.10 -> 1.12)
* 16:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set import sources on en.wikiversity [[gerrit:250709]] (duration: 00m 27s)
* 16:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on ru.wikipedia.org [[gerrit:252911]] (duration: 00m 26s)
* 16:19 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Editaton contra la violencia hacia las mujeres throttle rule [[gerrit:253541]] (duration: 00m 27s)
* 16:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 50% of new accounts on eswiki [[gerrit:250472]] (duration: 00m 26s)
* 16:02 hashar: Zuul-merger deployment aborted / uncomplte (I hate puppet)
* 16:02 bblack: re-enabling pybal on lvs400[12] (upgraded to 1.12)
* 15:36 cmjohnson1: reseating pem 0 on cr1-eqiad
* 15:30 mobrovac: restbase end deploy of e749f6ff
* 15:20 mobrovac: restbase start deploy of e749f6ff
* 14:59 bblack: stopping pybal on lvs300[12], fallback to lvs300[34] (pybal 1.10 vs 1.12)
* 14:55 godog: swift eqiad-prod: set ms-be1019 / ms-be1020 / ms-be1021 weight 1500
* 14:51 bblack: stopping pybal on lvs200[123], fallback to lvs200[456] (pybal 1.10 vs 1.12)
* 14:37 bblack: stopping pybal on lvs400[12], fallback to lvs400[34] (pybal 1.10 vs 1.12)
* 12:32 akosiaris: depool restbase1002
* 09:54 godog: nodetool decommission on restbase2002
* 06:18 gwicke: restbase cassandra: Increased tombstone_threshold from 0.02 to 0.1 for wikipedia and wikimedia html and data-parsoid to reduce single-sstable compaction rate.
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 26s)
* 00:21 logmsgbot: krinkle@tin Synchronized wmf-config/jobqueue-codfw.php: (no message) (duration: 00m 26s)
* 00:16 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: REL1_26 knocks on the door - https://gerrit.wikimedia.org/r/#/c/252399/ (duration: 00m 27s)
 
== 2015-11-16 ==
* 22:47 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.6/includes/Revision.php: Don't claim model validation failed if the content couldn't be loaded (duration: 00m 26s)
* 22:45 gwicke: rolling restbase restart to apply https://gerrit.wikimedia.org/r/#/c/253468/
* 21:52 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Don't log sysops with short passwords (duration: 00m 27s)
* 21:47 mobrovac: restbase end deployment of 54bc0517
* 21:38 mobrovac: restbase start deployment of 54bc0517
* 21:26 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log username of users with short passwords (duration: 00m 26s)
* 21:23 subbu: finished deploying parsoid 3a6f3b9e
* 21:21 mobrovac: restbase canary deploy to restbase1001 of 54bc0517
* 21:07 subbu: synced fresh code; restarting parsoid on wtp1005 as a canary
* 21:06 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Log privileged users with short passwords. Set minimum password length of 10 for wikitech. (duration: 00m 26s)
* 21:03 subbu: starting parsoid deploy
* 19:52 anomie: Running cleanupBlocks.php on arwiki, eswikibooks, eswikinews, and testwiki for [[phab:T118625]]
* 19:23 RoanKattouw: Deployed patch for T116095
* 16:14 logmsgbot: thcipriani@tin Synchronized wmf-config/squid.php: SWAT: wgHTCPRouting: use separate address for upload [[gerrit:249121]] (duration: 00m 26s)
* 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable first QuickSurveys survey [[gerrit:253038]] (duration: 00m 26s)
* 14:40 godog: carbon daemon bounce on graphite1001
* 13:43 bblack: reinstalling lvs100[45]
* 13:39 jynus: dropping ipblocks_old table from all wikis
* 13:19 bblack: reinstalling lvs1006
* 11:21 jynus: defragementing, upgrading, general maintenance to mysql at db1027. Expect lag, etc. (it is depooled)
* 10:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1015, depool db1027 for maintenance (duration: 00m 49s)
* 10:03 jynus: performing schema change on wikishared.bounce_records (x1)
* 09:39 godog: bootstrap restbase2001-c cassandra instance
* 09:34 jynus: adding missing tables to vewikimedia https://phabricator.wikimedia.org/rMW7389d7c69035c968553fbd2eaf1cf47038c4577c
* 09:19 _joe_: reimaging terbium
* 09:16 godog: swift eqiad-prod: add ms-be1019 / ms-be1020 / ms-be1021 weight 500
* 08:56 _joe_: rsyncing files from terbium to rutherfordium
* 08:10 akosiaris: powered off plutonium, removed it from puppet and salt. context: https://phabricator.wikimedia.org/T118586
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 29s)
 
== 2015-11-15 ==
* 18:03 jynus: disabling puppet, restarting mysql on es2008, es2009, es2010 for configuration check/upgrade
* 16:41 jynus: stopping eventlogging_sync for manual resync on dbstore1002 and db1047
* 15:33 logmsgbot: reedy@tin Synchronized wmf-config/: noop after chmod (duration: 00m 27s)
* 14:31 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: revert last, no file created on fluorine (duration: 00m 27s)
* 14:28 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily enable error log. Will revert in a few minutes (duration: 00m 27s)
* 14:19 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Pull in a few fixes and improvements for BounceHandler (duration: 01m 12s)
* 12:17 jynus: restarting mysql at db1015 to finish upgrade/maintenance process
* 12:10 jynus: uploading upgraded mariadb package for trusty, too
* 09:02 _joe_: rebooting krypton
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 06m 07s)
* 01:39 logmsgbot: reedy@tin Purged l10n cache for 1.27.0-wmf.1
* 01:36 ostriches: deploying scap master@54d93b3
 
== 2015-11-14 ==
* 23:31 Reedy: dsh delete php-1.26wmf22 23 and 24
* 23:06 Reedy: dsh delete php-1.27.0-wmf.4/cache/l10n/*
* 23:06 Reedy: dsh delete php-1.27.0-wmf.3/cache/l10n/*
* 23:04 Reedy: dsh delete php-1.27.0-wmf.2/cache/l10n/*
* 23:04 Reedy: dsh delete php-1.27.0-wmf.1/cache/l10n/*
* 23:03 Reedy: dsh delete php-1.26wmf24/cache/l10n/*
* 23:01 Reedy: dsh delete php-1.26wmf23/cache/l10n/*
* 23:00 Reedy: dsh delete php-1.26wmf22/cache/l10n/*
* 22:58 Reedy: dsh delete php-1.26wmf21
* 22:57 Reedy: dsh delete php-1.26wmf20
* 22:57 Reedy: dsh delete php-1.26wmf19
* 22:56 Reedy: dsh delete php-1.26wmf18
* 22:55 Reedy: dsh delete php-1.26wmf17
* 22:54 Reedy: dsh delete php-1.26wmf16
* 22:51 logmsgbot: reedy@mira Synchronized phpunit.xml: (no message) (duration: 00m 29s)
* 22:49 logmsgbot: reedy@tin Synchronized php-1.26wmf24/: purge l10n cache... test if deletes are propogated (duration: 01m 12s)
* 22:43 Reedy: deleted more old l10ncaches from tin staging 1.27 wmf1-4
* 22:37 logmsgbot: reedy@tin Synchronized phpunit.xml: (no message) (duration: 00m 30s)
* 22:26 logmsgbot: reedy@tin Synchronized php-1.26wmf24: remove old localisation cache (duration: 01m 38s)
* 22:22 logmsgbot: reedy@tin Synchronized php-1.26wmf23: remove old localisation cache (duration: 01m 40s)
* 22:10 logmsgbot: reedy@tin Synchronized phpunit.xml: noop test (duration: 01m 03s)
* 20:51 Reedy: Deleted php-1.26wmf11, php-1.26wmf12 and php-1.26wmf13 from mira /srv/mediawiki-staging
* 20:40 logmsgbot: reedy@tin Synchronized php-1.27.0-wmf.6/extensions/BounceHandler/: Email header logging on unsubscribe (duration: 00m 53s)
* 13:04 anomie: Ran cleanupBlocks.php on ruwiki for [[phab:T118625]]
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 58s)
* 01:58 ori: restarted HHVM on all app servers
 
== 2015-11-13 ==
* 23:30 logmsgbot: ori@tin Synchronized rpc/RunJobs.php: I5e15ec9fb: Disable ChronologyProtector for rpc/RunJobs.php (duration: 00m 30s)
* 23:27 awight: Reenabled fundraising campaigns outside of France
* 23:09 awight: Disabled Fundraising campaigns due to PayPal outage
* 21:23 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
* 21:17 YuviPanda: bounced etherpad again
* 20:39 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
* 20:38 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s)
* 20:35 logmsgbot: maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s)
* 20:30 logmsgbot: maxsem@tin Synchronized portals/: Bump to master (duration: 00m 39s)
* 20:19 robh: cr2-xe-1/3/0 optic install completed (detects fine)
* 20:18 robh: cr1-ulsfo:xe-0/0/2  optic install completed (detects fine)
* 20:17 robh: cr1-ulsfo:xe-0/0/2 (spare port) xfp optic detects fine, now installing other spare into cr2-xe-1/3/0
* 20:14 robh: installing xfp optics into cr1-ulsfo:xe-0/0/2 (spare port)
* 18:04 bblack: added neodymium to salt term in common-infrastructure4 on cr[12]-eqiad
* 17:37 csteipp: applied and reverted patch for T118032 (lots of exceptions on deploy)
* 16:16 jynus: restarting mysql on db2056, db2063 and db2064 for regular upgrade
* 15:16 godog: bootstrap restbase2001-b cassandra instance
* 15:03 jynus: restarting db2058, db2065 for regular mysql upgrade
* 14:43 jynus: restarting db2059, db2066 for regular mysql upgrade
* 11:42 jynus: restarting db2060 mysql for upgrade
* 09:20 jynus: performing table optimization on db1015, expect some lag while depooled
* 08:37 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 at 100%, depool db1015 (duration: 01m 31s)
* 06:58 _joe_: trying to manually syncing the ubuntu mirrors, since they are corrupted at the moment
* 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 21s)
* 01:27 YuviPanda: bounced mw1045 HHVM for ebernhardson
* 01:15 logmsgbot: catrope@tin Finished scap: I totally forgot that the SWAT contained i18n changes (duration: 26m 46s)
* 00:48 logmsgbot: catrope@tin Started scap: I totally forgot that the SWAT contained i18n changes
* 00:28 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/QuickSurveys: SWAT (duration: 00m 30s)
* 00:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/Flow: SWAT (duration: 00m 32s)
* 00:23 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.6/extensions/CentralNotice: SWAT (duration: 00m 31s)
* 00:16 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Remove hardcoded max upload size for UploadWizard (duration: 00m 30s)
* 00:15 awight: Banner impressions loader service restored
* 00:14 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Turn off language detection user test for CirrusSearch (duration: 00m 30s)
* 00:04 awight: update DjangoBannerStats from 08ffecf412a6a9ff3e173dded6814092834c7d1e to 26b48add5c6f70b395a3195c401a12612c626796
 
== 2015-11-12 ==
* 23:41 awight: update DjangoBannerStats from 9cc2acba7f6b46e243ada0fac6c48c9a49460a07 to 08ffecf412a6a9ff3e173dded6814092834c7d1e
* 23:05 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Monolog: wrap channel handlers in a WhatFailureGroupHandler (b08aaf4) (duration: 00m 29s)
* 22:46 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.6
* 22:42 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/TranslationNotifications/TranslationNotificationJob.php: hotfix T116960 (duration: 00m 30s)
* 21:21 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/resources/lib/jquery.i18n/src/jquery.i18n.language.js: T118242 - Unbreak IE8 (duration: 00m 30s)
* 21:10 chasemp: reboot analytics1040 it locked up
* 19:37 awight: Update DjangoBannerStats from 12e819a04a40ee6fab5dd55fcaf072661df31106 to 9cc2acba7f6b46e243ada0fac6c48c9a49460a07
* 19:24 awight: banner impressions job disabled
* 19:11 awight: Disabled WMDE and WMF Fundraising campaigns to do banner impression maintenance
* 19:05 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Set throttle exception for University of Haifa wiki event [[gerrit:252010]] (duration: 00m 29s)
* 18:21 logmsgbot: maxsem@tin Synchronized portals/: The rest of portals (duration: 00m 31s)
* 17:25 subbu: finished parsoid deploy sha 392e25eb
* 17:02 subbu: starting parsoid deploy
* 16:29 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy QuickSurveys to English Wikipedia [[gerrit:252606]] (duration: 00m 29s)
* 16:28 ostriches: gerrit: reloading replication plugin to pick up new config
* 16:11 andrewbogott: depooled T118469.  https://phabricator.wikimedia.org/T118469
* 16:08 bd808: cleaned up stale branch checkouts on mw1041: php-1.26wmf16 php-1.26wmf17 php-1.26wmf18 php-1.26wmf19 php-1.26wmf20 php-1.26wmf21
* 16:02 mobrovac: restbase end deploy of e5e45d11ad3f
* 15:48 mobrovac: restbase start deploy of e5e45d11ad3f
* 15:46 mobrovac: restbase canary deploy on rb1001 of e5e45d11ad3f
* 15:39 jynus: powercycling mw1041
* 15:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 after maintenance (duration: 00m 29s)
* 10:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.6/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 00m 31s)
* 09:53 jynus: rebooting/upgrading/restarting mariadb on db1035 after maintenance finished
* 09:52 jynus: uploading new mariadb package linked to openssl instead of yassl
* 09:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/UniversalLanguageSelector: I9206a79: Only use jQuery.tipsy for undo popover (duration: 01m 15s)
* 09:39 moritzm: corrected apt sources on db2057, db2058, db2062, db2065, db2066, db2067, db2068, db2069
* 08:42 moritzm: restarted salt-master on palladium
* 07:35 _joe_: restarting pybal on lvs1004
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 56s)
* 01:02 gwicke: restbase: finished full deploy of restbase-deploy ce4abe784
* 00:50 gwicke: restbase: starting full deploy of restbase-deploy ce4abe784
* 00:37 gwicke: restbase: canary deploy of restbase-deploy ce4abe784 on restbase1001
* 00:32 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in QuickSurveys config (duration: 00m 31s)
 
== 2015-11-11 ==
* 23:48 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
* 23:44 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/includes/jobqueue/aggregator/: sync https://gerrit.wikimedia.org/r/#/c/252588/ (duration: 00m 32s)
* 21:13 subbu: finished deploying parsoid sha 7ca999c1
* 21:06 subbu: synced new code + restarted parsoid on wtp1001 as canary; monitoring graphs for a little bit
* 21:02 subbu: starting parsoid deploy
* 20:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
* 19:51 hoo: Started rebuildItemsPerSite for wikidatawiki on mw1152. Feel free to kill, should it cause troubles.
* 19:49 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.6
* 19:37 twentyafterfour: deploying 1.27.0-wmf.6 to group1
* 19:09 YuviPanda: nfs-exports on labstore1001 failed because of http failure of wikitech
* 19:05 YuviPanda: started nfs-exports service on labstore1001
* 18:18 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2067 after maintenance (duration: 00m 29s)
* 16:46 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II [[gerrit:252349]] (duration: 00m 30s)
* 16:45 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I [[gerrit:252349]] (duration: 00m 30s)
* 16:34 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Restore satisfaction schema and fix the performance issue that it had part II [[gerrit:252347]] (duration: 00m 30s)
* 16:33 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.6/extensions/WikimediaEvents/WikimediaEvents.php: SWAT: Restore satisfaction schema and fix the performance issue that it had part I [[gerrit:252347]] (duration: 00m 40s)
* 16:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: First QuickSurvey for reader segmentation research - external survey [[gerrit:251133]] (duration: 00m 30s)
* 16:02 mutante: uploaded jenkins 1.625.2 to apt.wm.org, upgrading on gallium
* 15:47 moritzm: restarted kafka broker on kafka1022
* 15:34 moritzm: stopping kafka broker on kafka1022 (to enable ferm)
* 15:31 moritzm: restarted kafka broker on kafka1020
* 15:25 moritzm: stopping kafka1020 to enable ferm
* 15:22 moritzm: restarted kafka1018 to enable ferm
* 15:19 moritzm: stopping kafka1018 to enable ferm
* 15:04 moritzm: disabled puppet on kafka1018, kafka1020, kafka1022 (for enabling ferm)
* 14:18 godog: start cassandra instance a on restbase2001
* 13:54 mobrovac: restbase deployed 0d961a2 (deploy end)
* 13:42 mobrovac: restbase deploying 0d961a2
* 11:28 godog: reimage restbase2001.codfw.wmnet - take #2
* 10:16 jynus: disabling puppet and restarting mysql on db2067
* 10:16 godog: reimage restbase2001.codfw.wmnet
* 10:07 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2067 for maintenance (not db1067) (duration: 00m 30s)
* 10:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1067 for maintenance (duration: 00m 30s)
* 09:57 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: really fix T118372 this time (duration: 00m 29s)
* 09:43 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.6/extensions/OAuth/: oauth login broken by php error. This fixes T118372 (duration: 00m 31s)
* 08:51 jynus: importing wmf-mariadb10_10.0.22-1_amd64.deb wmf-mysql57_5.7.9-1_amd64.deb on carbon
* 04:19 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: rm some old log channels (duration: 00m 31s)
* 03:23 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.6/includes/cache/MessageBlobStore.php: Logging for T93800 (duration: 00m 31s)
* 02:51 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.5/includes/Title.php: Deploy I47517021471 which was merged earlier today but forgotten (duration: 01m 58s)
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 07m 01s)
* 01:48 mutante: repooled mw1083 (T116184)
* 00:58 mutante: added jenkins_1.625.1 to APT repo. precise-wikimedia
* 00:50 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.6
* 00:38 logmsgbot: twentyafterfour@tin Finished scap: grr: sync 1.27.0-wmf.6 (duration: 55m 05s)
 
== 2015-11-10 ==
* 23:42 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
* 23:41 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.7rNhS6PIRi" ' returned non-zero exit status 1 (duration: 00m 09s)
* 23:41 logmsgbot: twentyafterfour@tin Started scap: grr: sync 1.27.0-wmf.6
* 23:40 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.yIUW9abUwO" ' returned non-zero exit status 1 (duration: 00m 15s)
* 23:39 logmsgbot: twentyafterfour@tin Started scap: trying again: sync 1.27.0-wmf.6
* 23:23 ori: ran sudo chgrp -R wikidev /srv/mediawiki-staging/php-1.27.0-wmf.6 on tin
* 22:50 ejegg: rolled back paymentswiki from 3730cff201e314fc54ad76fbc54f33f5b032def5 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
* 22:43 logmsgbot: twentyafterfour@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.sEpytaqlt1" ' returned non-zero exit status 1 (duration: 01m 42s)
* 22:41 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.6
* 21:51 hashar: installed Zuul package on scandium.eqiad.wmnet . zuul-merger not running
* 21:33 ejegg: updated paymentswiki from 4baa5a66a4510414d6b43b59f1b1cda2341c17fd to 3730cff201e314fc54ad76fbc54f33f5b032def5
* 21:16 ejegg: enabled banner_history module on CiviCRM
* 21:15 mutante: scb1001 - mobile endpoint health CRIT
* 21:15 mutante: mira unmerged changes in mw-staging
* 21:14 mutante: restbase2001: cassandra CQl refused
* 21:14 mutante: aqs1001-1003: CRITICAL: Test Get aggregate page views
* 21:12 mutante: tungsten - signed new puppet certs, re-installed
* 20:48 moritzm: kafka re-enabled on kafka1013
* 20:42 moritzm: stopping kafka broker on kafka1013 to activate ferm
* 20:26 ottomata: starting kafka broker on kafka1012 after applying base ferm
* 20:14 ottomata: stopping kafka broker on kafka1012 to apply base ferm
* 20:09 moritzm: disabled puppet on kafk1012/1013 in preparation of ferm activation
* 20:08 _joe_: restarting jobchron across the cluster
* 18:41 ottomata: powercycling analytics1031
* 16:51 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/252231/ (duration: 00m 35s)
* 16:45 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 00m 34s)
* 16:44 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/252204/ (duration: 00m 34s)
* 16:41 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/250000/ (duration: 00m 34s)
* 16:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250471/ (duration: 00m 34s)
* 16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: https://gerrit.wikimedia.org/r/#/c/252172/ (duration: 00m 35s)
* 16:22 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Wikidata/Wikidata.php: https://gerrit.wikimedia.org/r/#/c/252227/ (duration: 00m 45s)
* 15:51 mutante: restarted krypton
* 15:32 godog: upgrade rsync on ms-be1* per T93587
* 14:02 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 3000
* 10:00 jynus: performing mysql maintenance of db1035 (depooled from production). Replication delay and reboots are expected.
* 09:46 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 2000
* 09:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1035 for maintenance (duration: 00m 36s)
* 06:13 ejegg|away: updated fundraising tools from f2f571903d32ec73aeab73ce58c6bee71717213b to 182f217ea20a496fe0f9737447d04ed9a3bccde8
* 03:12 ottomata: restarted statsv on hafnium just in case it stopped after kafka1013 broker crash and restart (ping ori :) )
* 03:10 ottomata: kafka preferred-replica-election to bring kafka1013  back as leader for its partitions
* 03:00 mutante: started kafka on kafka1013
* 02:55 mutante: tried to start kafka on kafka1018
* 02:31 mutante: switched people.wm.org to new backend. please use "rutherfordium.eqiad.wmnet" instead of terbium now. all files have been rsynced already
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)
* 00:13 hoo: Attached Richardelainechambers@commonswiki to the global account of the same name
* 00:06 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: Disable microsecond timestamps on all loggers (bf61bfc) (duration: 00m 34s)
 
== 2015-11-09 ==
* 23:55 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Disable cirrus writes to labsearch, the machine cant take the load and some jobs are timing out (duration: 00m 34s)
* 23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Enable CirrusSearch writes to labsearch for all but enwiki and dewiki (duration: 00m 35s)
* 22:49 ottomata: rebooting analytics1032: https://phabricator.wikimedia.org/T118175
* 21:52 bearND: MobileApps deployed sha1 6c63984
* 19:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1060 after performance patch (duration: 00m 35s)
* 18:42 jynus: restarting mysql in db1060 to test new performance configuration
* 17:55 bblack: upgrading pybal (1.10 -> 1.12) on lvs200[456].codfw.wmnet
* 16:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250454/ (duration: 00m 34s)
* 16:44 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init: https://gerrit.wikimedia.org/r/#/c/251972/ (duration: 00m 34s)
* 16:40 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251674/ (duration: 00m 35s)
* 16:35 logmsgbot: krenair@tin Synchronized docroot/wikipedia.org/apple-app-site-association: https://gerrit.wikimedia.org/r/#/c/250897/ (duration: 00m 34s)
* 16:30 matt_flaschen: Ran mwscript extensions/Flow/maintenance/FlowUpdateBetaFeaturePreference.php --wiki=wikidatawiki
* 16:24 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251677/ (duration: 00m 35s)
* 16:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow/modules/mw.flow.Initializer.js: https://gerrit.wikimedia.org/r/#/c/251560/ (duration: 00m 44s)
* 16:10 _joe_: removing old builds from copper
* 15:45 godog: reimage graphite1002
* 15:06 ottomata: running kafka preferred-replica-election
* 15:02 ottomata: starting kafka broker on kafka1014
* 14:56 ottomata: stopping kafka broker on kafka1014
* 14:48 godog: swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 1000
* 14:40 ottomata: restarting eventlogging to see if it is ok after enabling firewall rules on kafka1014
* 14:36 _joe_: reducing /tmp size on copper by shrinking the logical volume
* 13:16 godog: nodetool decomission restbase2001
* 13:13 mobrovac: restbase finished deploying ae2a44f
* 12:59 mobrovac: restbase start deploying ae2a44f
* 12:56 hashar: restarting Jenkins to refresh the cli-shutdown.groovy script -- https://gerrit.wikimedia.org/r/251935 (https://phabricator.wikimedia.org/T118064)
* 12:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depooling db1060 again (duration: 01m 13s)
* 11:16 hashar: Jenkins back and apparently happy
* 11:09 hashar: Upgrading Jenkins from LTS 1.609.3 to LTS 1.625.1 https://phabricator.wikimedia.org/T118157
* 10:53 dcausse: resuming writes to elasticsearch indices in eqiad (test)
* 10:02 _joe_: repooled mw1061
* 09:29 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 3000
* 09:19 dcausse: err (1007 no 1008): restarting elastic on elastic1007.eqiad.wmnet (test)
* 09:18 dcausse: restarting elastic on elastic1008.eqiad.wmnet (test)
* 09:10 dcausse: freezing elasticsearch indices in eqiad (test)
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s)
 
== 2015-11-08 ==
* 20:43 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ConfirmEdit/SimpleCaptcha/Captcha.php: https://gerrit.wikimedia.org/r/#/c/251673/ (duration: 00m 34s)
* 20:21 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources: (no message) (duration: 00m 34s)
* 20:14 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/resources/src: I24d9b16ed: Rework the Preferences to prevent FOUC (T115692) (duration: 00m 34s)
* 17:10 _joe_: powercycled ms-be1009, unresponsive to ping, console unconneptable
* 13:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1060 (duration: 00m 34s)
* 13:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 01m 29s)
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 21s)
 
== 2015-11-07 ==
* 19:50 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/251739/ (duration: 00m 35s)
* 17:54 ejegg|away: updated SmashPig from 5860ef47790010d65981822f30914679df06ae14 to 55c2dbc8cd36cfb89d7e17b75b840ca2e023653e
* 17:38 awight: more syntax fixing
* 17:35 awight: fix SmashPig failmail syntax
* 17:29 awight: point SmashPig failmail at ejegg|away and awight
* 17:28 jynus: restarting labsdb1004 mysql to test mariadb upgrade
* 17:09 jynus: restarting db2070 mysql to test mariadb upgrade
* 17:01 jynus: restarting es2010 mysql to test mariadb upgrade
* 15:17 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Return db weights on s2 api back to normal (duration: 00m 34s)
* 11:24 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1054 (duration: 00m 34s)
* 11:02 jynus: setting SET GLOBAL use_stat_tables = 0; on db1060
* 10:45 jynus: restarting db1054 (I may need to do it several times)
* 10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1054 (duration: 00m 35s)
* 10:27 jynus: reverting last schema change
* 10:08 jynus: performing schema change on db1054 (zhwiki)
* 05:00 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 34s)
* 04:59 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: (no message) (duration: 01m 09s)
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 12s)
* 00:23 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)
* 00:05 ejegg: updated SmashPig from ac9772765287e79178e07ddfcc7e532d2f82b6d0 to 5860ef47790010d65981822f30914679df06ae14
* 00:04 logmsgbot: maxsem@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaMaintenance/getPageCounts.php: (no message) (duration: 00m 34s)
 
== 2015-11-06 ==
* 23:48 ejegg: updated SmashPig listener from afefccff0f8d5e0b37044adff964c6cd913e1d80 to ac9772765287e79178e07ddfcc7e532d2f82b6d0 and set failmail target back to fr-tech
* 23:34 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Enable DynamicPageList on officewiki (duration: 01m 03s)
* 22:42 legoktm: mwscript updateSpecialPages.php --wiki=enwiki --only=GadgetUsage
* 19:22 andrewbogott: disabled puppet on holmium while testing and staging an update for bug T117610
* 19:13 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I920627cf8b: Add rdb1007 as fallback aggregator (duration: 00m 35s)
* 18:54 logmsgbot: ori@tin Synchronized wmf-config/jobqueue-eqiad.php: I9e5be66680: Add rdb1007 to jobqueue pool (duration: 00m 35s)
* 18:19 ori: running sync-common on mw1083
* 17:45 logmsgbot: demon@tin Synchronized multiversion/checkoutMediaWiki: no-op outside tin; just for completeness (duration: 00m 36s)
* 15:29 paravoid: draining esams for network maintainance (deployed about 7mins ago)
* 14:52 moritzm: uploaded debdeploy 0.0.9 to apt.wikimedia.org
* 14:31 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool all available db servers in codfw (duration: 00m 49s)
* 13:58 godog: bounce ganglia-monitor-aggregator-instance ID=2027 on install2001, debugging missing ms-be2020
* 13:38 godog: swift codfw-prod: ms-be2016 / ms-be2018 / ms-be2020 weight 2000
* 13:36 bblack: cp4007 repooled - T117746
* 13:16 bblack: upgrading pybal 1.10 -> 1.12 on lvs300[34].esams.wmnet
* 13:11 bblack: re-enabled aws-d-eqiad in librenms
* 13:09 bblack: upgrading pybal 1.10 -> 1.12 on lvs4003
* 12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10G line card
* 12:18 mark: Swapped cr2-knams fpc 1 mic 1 for a new 2x 10
* 12:18 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 online
* 12:10 mark: mark@cr2-knams> request chassis mic fpc-slot 1 mic-slot 1 offline
* 11:31 jynus: stopping mysql and cloning db2042 -> db2070, db2054 -> db2068, db2053 -> db2067, db2049 -> db2056
* 11:12 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2042, db2049, db2053 and db2054 for cloning (duration: 00m 34s)
* 09:09 jynus: Stopping mysql and cloning db2044 -> db2065, db2048 -> db2069, db2045 -> db2066
* 08:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db20 34,42,35,38,39,40. New: 58,64,60,61,62,59. Depool: 44,45,48 (duration: 01m 10s)
* 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 05m 56s)
* 01:43 logmsgbot: krenair@mira Synchronized README: done (duration: 00m 34s)
* 01:38 logmsgbot: krenair@mira Synchronized README: test (duration: 00m 34s)
* 01:33 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251446/ (duration: 00m 35s)
* 01:30 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (forgot to apply after fetching) (duration: 00m 34s)
* 01:28 logmsgbot: bd808@tin Synchronized wmf-config/InitialiseSettings.php: Touch (duration: 00m 35s)
* 01:26 logmsgbot: bd808@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/249313/ (duration: 00m 34s)
* 01:10 logmsgbot: krenair@tin Synchronized multiversion/defines.php: https://gerrit.wikimedia.org/r/#/c/251447/ (duration: 00m 34s)
* 01:00 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 34s)
* 00:59 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: https://gerrit.wikimedia.org/r/#/c/251301/ https://gerrit.wikimedia.org/r/#/c/251306/ and https://gerrit.wikimedia.org/r/#/c/251307/ (duration: 00m 35s)
* 00:34 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/251317/ (duration: 00m 35s)
* 00:18 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/251416/ (duration: 00m 37s)
 
== 2015-11-05 ==
* 23:34 bd808: Decreased replica count of logstash-2015.10.13 and logstash-2015.10.14 to free disk space on cluster
* 23:30 bd808: Logging volume into ELK cluster down dramatically; investigating
* 23:16 logmsgbot: ori@tin Synchronized wmf-config/logging.php: Ieb8c602a: monolog: Ensure that context data added by WebProcessor is utf-8 safe (duration: 00m 36s)
* 22:35 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.5/extensions/WikimediaEvents: Ic99ac31f740956: Log backend response time on edit requests (duration: 00m 35s)
* 21:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: sync 1.27.0-wmf.5 to group2
* 20:17 logmsgbot: demon@tin Finished scap: no changes, testing permissions on mira co-master (duration: 07m 29s)
* 20:10 logmsgbot: demon@tin Started scap: no changes, testing permissions on mira co-master
* 20:07 _joe_: chown mwdeploy:wikidev recursively on mira for /srv/mediawiki-staging
* 19:37 logmsgbot: twentyafterfour@tin Finished scap: Sync everything just to be sure (duration: 22m 01s)
* 19:15 logmsgbot: twentyafterfour@tin Started scap: Sync everything just to be sure
* 18:58 logmsgbot: demon@tin Synchronized README: no-op, testing new scap code (duration: 00m 19s)
* 18:52 ostriches: scap: deploying master@b44c268
* 18:05 jynus: schema change on x1 - flowdb
* 17:47 ostriches: 17:45:13 Synchronized wmf-config/: Remove +superprotect, I579c11a2 (duration: 00m 18s)
* 17:44 jynus: Deploying schema change on officewiki - flow (s3)
* 17:11 godog: nodetool decommission on praseodymium
* 14:12 jynus: reinstalling db2056.codfw.wmnet
* 13:57 jynus: shuting down mysql and cloning db2042 -> db2062, db2038 -> db2059, db2039 -> db2060, db2040 -> db2061
* 13:43 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Repool db2051; Depool db2042, 38, 39, 40 for cloning (duration: 00m 18s)
* 12:33 _joe_: manually disabling mod_cgi on terbium
* 09:56 godog: run removenode on cerium.eqiad.wmnet -- decomission was missed before reimaging
* 09:33 jynus: stopping mysql and cloning db2034 -> db2062, db2035 -> db2063, db2051 -> db2058
* 09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Pool db2048, db2049, db2050,db2055, db2057, db2063. Depool db2034, db2035, db2051 (duration: 00m 17s)
* 08:05 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Nov  5 08:05:13 UTC 2015 (duration 5m 12s)
* 03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-05 03:21:45+00:00
* 03:15 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 17s)
* 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-05 02:47:20+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 10m 31s)
* 02:17 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I3c397e892e: Only set $wgDisableOutputCompression to 'true' on the scalers (duration: 00m 18s)
* 02:04 Krenair: Someone has left tin:/srv/mediawiki-staging/php-1.27.0-wmf.5 in a mess, see `git log origin/wmf/1.27.0-wmf.5..HEAD --oneline`. Note there is one commit waiting to be merged on tin (https://gerrit.wikimedia.org/r/#/c/251168/) that hasn't been yet because of this.
* 00:13 ejegg: updated fundraising tools from e1b60fa2c258fd4ff55905b03a4d8886132278c1 to f2f571903d32ec73aeab73ce58c6bee71717213b
 
== 2015-11-04 ==
* 23:56 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: ced9a62893c (duration: 00m 17s)
* 23:55 ejegg: updated CiviCRM from 827faf37b1b8f4ade6f8d03077c376a09d9525c7 to 230cebb4a3b46a75fef758d46f4db22de29286cb
* 23:09 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: 33d378bee6c4ac (duration: 00m 17s)
* 22:33 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.5/maintenance/findOrphanedFiles.php: fb43e95858b9 (duration: 00m 18s)
* 21:36 ejegg: enabled Adyen payments listener
* 21:17 subbu: finished deploying parsoid sha 04893a18
* 21:05 subbu: synced code + restarted parsoid on wtp1002 as a canary
* 21:01 subbu: starting parsoid deploy
* 20:45 ejegg: disabled Adyen payment listener
* 20:36 ejegg: re-enabled Adyen payment listener with updated credentials
* 20:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.5
* 18:37 mutante: correction. ^ that was mw1159
* 18:35 mutante: mw1159: power cycled - crashed, no console output
* 18:23 logmsgbot: bd808@tin Synchronized php-1.27.0-wmf.5/includes/debug/logger/MonologSpi.php: MonologSpi: add support for customizing Monolog\Logger instances (59eed42) (duration: 00m 17s)
* 18:23 mutante: ms-be2016-2020 - accepted salt keys, hafnium.wm.org - deleted salt key - still unaccepted: capella, haedus, rhodium
* 17:57 mutante: ms-be2020 - added to puppet
* 17:54 mutante: ms-be2021 - fixing puppet certs after reinstall
* 17:51 papaul: ms-be2021 installation complete
* 17:45 papaul: reinstalling ms-be2021
* 17:20 ejegg: updated SmashPig from be9bf87c5583bca75616284f6a4d16ff8ebd3679 to afefccff0f8d5e0b37044adff964c6cd913e1d80
* 17:12 chasemp: rebooting cp4007.mgmt.ulsfo.wmnet as it's uresponsive
* 17:11 bblack: cp4007 depooled in pybal + confctl
* 17:10 papaul: ms-be2020 installation complete
* 16:56 ejegg: disabled adyen payments listener
* 16:30 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250971/ (duration: 00m 18s)
* 16:29 bd808: restarted elasticsearch on logstash1004; it fell out of the cluster during ferm firewall application
* 16:28 jynus: db2056 clone failed due to wrong partitioning, needs reinstall. Cloning db2049 -> db2063 instead
* 16:01 godog: reimage cerium for multi-instance
* 15:22 _joe_: moving email batch, flagged reviews, update article count off of terbium
* 15:20 jynus: syncing sqldata on db2049 -> db2056, db2050 -> db2057, eta 2-4 hours
* 15:15 _joe_: moved translationnotification, updatetranslationstats, pagetriage from terbium to mw1152
* 14:55 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2049 ans db2050 for maintenance (duration: 00m 17s)
* 14:30 jynus: cloning sqldata from db2048 to db2055
* 14:04 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Depool db2048 for maintenance (duration: 00m 18s)
* 13:03 bblack: upgrading pybal (1.10 -> 1.12) on lvs4004.ulsfo.wmnet (inactive low-traffic LVS @ ulsfo)
* 11:59 _joe_: moved refreshlink jobs off terbium
* 11:32 _joe_: migrating wikidata jobs off of terbium
* 11:07 jynus: testing new mariadb-server version on db2056.codfw.wmnet
* 09:25 godog: bootstrap cassandra xenon-b.eqiad.wmnet
* 08:47 jynus: reduced durability on dbstore1002 and db1047 to improve performance
* 08:27 _joe_: moved parser cache purging off of terbium
* 08:25 _joe_: moved clean_uploadstash to mw1152 from terbium
* 06:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Nov  4 06:44:23 UTC 2015 (duration 44m 22s)
* 05:52 logmsgbot: demon@tin Finished scap: using wfLoadExtension() for ContentTranslation, shut up warning spam (duration: 49m 22s)
* 05:03 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
* 05:02 logmsgbot: demon@tin scap failed: CalledProcessError Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/.svn/lock', '--exclude=**/.git/objects', '--exclude=**/.git/**/objects', '--exclude=**/cache/l10n/*.cdb', '--no-perms', 'tin.eqiad.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 24 (duration: 02m 45s)
* 04:59 logmsgbot: demon@tin Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam
* 04:54 ejegg|away: updated smashpig from c43db1742d5663f9d3685fd3e0a254ff72a29fa0 to be9bf87c5583bca75616284f6a4d16ff8ebd3679, changed credentials
* 03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-04 03:09:44+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 08s)
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-04 02:36:48+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 34s)
* 01:36 legoktm: ran delete from globaluser where gu_name="" limit 1 on centralauth. (T96233)
* 01:10 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/250874/ (duration: 00m 17s)
* 01:02 hoo: Manually recreated https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf using Zend php (instead of HHVM). See T117534.
* 00:25 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250869/ (duration: 00m 19s)
* 00:09 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250712/ (duration: 00m 18s)
* 00:03 logmsgbot: krenair@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/250851/ (duration: 00m 17s)
 
== 2015-11-03 ==
* 23:11 ejegg: deployed Adyen credentials change to payments wiki
* 22:36 ostriches: tin: scap now pointing to Phab repo instead of Gerrit.
* 21:59 logmsgbot: krenair@tin Synchronized README: rv, was just demonstrating for https://phabricator.wikimedia.org/T117574#1779384 (duration: 00m 17s)
* 21:42 logmsgbot: krenair@tin Synchronized README: (no message) (duration: 00m 18s)
* 21:21 mutante: ms-be2018,ms-be2019: sign puppet certs, initial run
* 20:45 csteipp: deployed patch for T109724 to wmf4/5
* 20:41 gwicke: restbase cassandra: switched local_group_wikipedia_T_parsoid_html to the Date-Tiered Compaction Strategy (DTCS)
* 20:38 gwicke: cleared cassandra snapshots on aqs servers
* 19:59 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.5
* 19:55 mutante: hafnium - revoke puppet cert, salt key for the .wikimedia.org name
* 19:54 mutante: hafnium - reboot into PXE for jessie reinstall
* 19:47 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.27.0-wmf.5 (duration: 30m 00s)
* 19:35 awight: update CRM from 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7 to 827faf37b1b8f4ade6f8d03077c376a09d9525c7
* 19:33 awight: update SmashPig from c431c8d77521270236c72532a50806b2e852cf7b to c43db1742d5663f9d3685fd3e0a254ff72a29fa0
* 19:17 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.27.0-wmf.5
* 19:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/objectcache/BagOStuff.php: 46dff75b5b: Make makeKeyInternal() limit more conservative (duration: 00m 18s)
* 19:06 twentyafterfour: deploying 1.27.0-wmf.5 to group0
* 18:34 ori: Moved statsv script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
* 18:29 ori: Moved navtiming script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage
* 17:13 moritzm: installed security updates on the internal ntpds (system-local ones to follow)
* 16:27 dcausse: restarting elastic on nobelium to check new settings
* 16:19 moritzm: installed unzip and audiofile security updates across the fleet
* 16:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sampling of CirrusSearchRequestSet log channel [[gerrit:250691]] (duration: 00m 18s)
* 16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on eswiki [[gerrit:250470]] (duration: 00m 18s)
* 14:19 jynus: disabling puppet on dbstore1002 and db1047 to debug replication issue
* 14:07 godog: manual invocation of /usr/local/bin/archive-instances on labmon1001 to test
* 13:33 jynus: migrating eventlogging_sync from terbium to db1047 and dbstore1002 (analytics-store)
* 12:15 _joe_: updated pybal for jessie to 1.12
* 11:00 godog: start cassandra-a on xenon
* 09:56 godog: reimage xenon.eqiad.wmnet
* 09:45 godog: decomission xenon.eqiad.wmnet from cassandra, pending conversion to multi-instance
* 05:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Nov  3 05:38:33 UTC 2015 (duration 38m 32s)
* 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-03 02:26:14+00:00
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 00s)
* 01:05 awight: enabled CRM module amazon_audit
* 01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/extensions/CentralAuth/includes/CentralAuthUser.php: Ia9673a448da81: User cache key (duration: 00m 17s)
* 01:03 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/: I58836a24b9e239f: User cache key (duration: 00m 21s)
* 00:37 logmsgbot: ebernhardson@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 19s)
* 00:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
* 00:34 logmsgbot: ebernhardson@tin Synchronized wmf-config/avro/CirrusSearchRequestSet.avsc: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s)
* 00:28 ebernhardson: sync-common on mw1017 for https://gerrit.wikimedia.org/r/240615
* 00:26 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241226/4 (duration: 00m 17s)
* 00:17 awight: update CRM from 77cd64317404b2fcec9500cfbb1c0b19e5846705 to 2828fdfbf791c6ca28d1f92725d1ec887b9fe4c7
* 00:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250590/ (duration: 00m 18s)
* 00:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: add www.webarchive.co.uk and *.unesco.org to server side upload whitelist (duration: 00m 18s)
* 00:06 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/246169/ (duration: 00m 18s)
 
== 2015-11-02 ==
* 23:11 RoanKattouw: Running convertAllLqtPages.php on ptwikibooks
* 22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Flow on ptwikibooks with English-only Topic namespace name (duration: 00m 18s)
* 22:38 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgFlowEnglishNamespaceOnly (duration: 00m 18s)
* 21:56 cwd: updated worldpay request timeout to 30s
* 21:31 _joe_: restarting parsoids in codfw
* 21:25 subbu: deployed parsoid f0d77afc
* 20:49 mutante: ms-be2017 - initial puppet run, fresh install
* 20:38 logmsgbot: csteipp@tin Synchronized php-1.27.0-wmf.4/includes/: (no message) (duration: 00m 22s)
* 20:38 robh: lawrencium having puppet stopped and reclaiming into spares per T117477 disregard any alerts
* 20:06 mutante: ms-be2016 - signing puppet certs, salt-key, initial run
* 20:05 mutante: started morebots (production-logbot job was gone)
* 19:46 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/includes/MobileFrontend.skin.hooks.php: Fix changing the license message key via the "MobileLicenseLink" hook (duration: 00m 17s)
* 19:14 gwicke: restbase cassandra: switching local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ to DTCS
* 19:00 matt_flaschen: Completed FlowFixLinks run on all Flow wikis
* 18:53 csteipp: deployed patches for T97897 & T115522
* 18:53 awight: updated CRM from 60c4a8e5fc4efb6ac38a520e652afe187e8c177b to 77cd64317404b2fcec9500cfbb1c0b19e5846705
* 18:48 andrewbogott: dist-upgrade and rebooting labvirt1005
* 18:41 Coren: manually starting service replicate-tools on labstore1001
* 18:10 awight: removed failing SmashPig job runner task
* 18:01 jynus: performing some fixes on labsdb1004 database. Slightly higher load on labsdb1005.
* 17:59 dcausse: restarting elastic on nobelium
* 17:43 awight: disabled Donations queue consume job
* 17:39 gwicke: restbase cassandra: switched local_group_wikisource_T_parsoid_html and local_group_wikibooks_T_parsoid_html to DTCS
* 17:09 bd808: Decreased replica count to 1 for logstash-2015.10.04 thru logstash-2015.10.12 to free cluster disk space; see T117438
* 17:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/249762/ (duration: 00m 19s)
* 17:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247088/ (duration: 00m 17s)
* 16:48 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
* 16:47 logmsgbot: krenair@tin Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ (duration: 00m 17s)
* 16:45 logmsgbot: krenair@tin Synchronized dblists/closed.dblist: https://gerrit.wikimedia.org/r/#/c/250416/ - close wikimania2014.wikimedia.org (duration: 00m 18s)
* 16:36 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249794/ - should be a noop for prod (duration: 00m 17s)
* 16:30 bd808: Deleted logstash-2015.10.03 index to free disk space on logstash1004; see T113571
* 16:28 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/249940/1 (duration: 00m 20s)
* 16:26 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/249935/ (duration: 00m 18s)
* 16:09 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250401/ (duration: 00m 18s)
* 14:43 ottomata: shutting down analytics1038 to run fsck on root filesystem.  Currently it is read only
* 14:36 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-br-fr_0.5.0+svn~57870-1
* 14:35 mobrovac: citoid deployed 70abb90
* 13:34 paravoid: draining ulsfo, T116928
* 12:19 mobrovac: mobileapps deploying 59e8c30
* 10:52 hashar: Restarting Jenkins to upgade Gearman plugin from 0.1.2 to 0.1.3 "Send node labels back on build completion"
* 10:10 akosiaris: powering off pollux. migrating to a VM
* 09:39 _joe_: moved tor exit node, update_special_pages off of terbium to mw1152
* 09:24 mobrovac: citoid restarted zotero on sca100x
* 09:07 jynus: deleting user_daily_contribs view on labs (it points to a non-existent table)
* 08:57 mobrovac: restbase deploying 8036232
* 05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Nov  2 05:33:07 UTC 2015 (duration 33m 6s)
* 02:25 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-02 02:25:42+00:00
* 02:22 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 44s)
 
== 2015-11-01 ==
* 16:59 gwicke: restarting cassandra on aqs1002; was out of heap space
* 05:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Nov  1 05:25:49 UTC 2015 (duration 25m 48s)
* 02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-01 02:24:46+00:00
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 36s)
 
== 2015-10-31 ==
* 08:57 dcausse: elastic1008 deleting /var/log/elasticsearch/production-search-eqiad_index_indexing_slowlog.log.[2-7]
* 05:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 31 05:15:43 UTC 2015 (duration 15m 42s)
* 02:54 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda4768 (duration: 00m 19s)
* 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-31 02:26:22+00:00
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 26s)
 
== 2015-10-30 ==
* 23:53 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ifd3543d99: Re-enabled sidebar cache per 47eb083a0fe4 (duration: 00m 17s)
* 23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/cache/MessageCache.php: Id9a27ba2bbd3 (duration: 00m 17s)
* 23:38 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/skins/Skin.php: Id9a27ba2bbd3 (duration: 00m 17s)
* 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Collection: b89cff0a1d: Updated mediawiki/core Project: mediawiki/extensions/Collection  092204bf333e65b2c749e4bc4a32fd2b0254089b (duration: 00m 18s)
* 23:37 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/DismissableSiteNotice: a20aa8a544: Updated mediawiki/core Project: mediawiki/extensions/DismissableSiteNotice  5b8f1cfac48704fd740eb61f873aabb600c4c5fb (duration: 00m 17s)
* 23:03 gwicke: fixed restbase labs install by deploying it; the code had gone missing
* 23:03 ori: Added MaxSem and gwicke to Gerrit Project Creators group
* 22:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/objectcache/ObjectCache.php: If12aedae7f: objectcache: Use singleton cache in newAccelerator() (duration: 00m 18s)
* 22:05 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
* 22:01 mutante: labvirt1010: salt-minion always in "stop/waiting", doesn't restart
* 21:54 mutante: labvirt1010 - start salt
* 21:41 ori: Removing MediaWiki.xhprof.* from graphite{1,2}001
* 21:41 logmsgbot: ori@tin Synchronized wmf-config/StartProfiler.php: I0bfa21b5: Disable xhprof profiling to relieve storage pressure on graphite (duration: 00m 18s)
* 19:07 K4-713: antifraud tweak on payments wiki (again)
* 18:56 K4-713: antifraud tweak on payments wiki
* 18:56 jynus: disabling puppet and restarting mysql on es2008, es2009 and es2010- downtime planned for 2 hours - no production impact
* 17:33 robh: bismuth has no console output, appears locked up in OS, rebooting via mgmt
* 17:22 logmsgbot: ori@tin Synchronized docroot and w: (no message) (duration: 00m 18s)
* 16:56 cmjohnson: reinstalling mw1083 https://phabricator.wikimedia.org/T116184
* 15:35 moritzm: updated db205[6-8] to the 3.19 kernel
* 14:38 chasemp: nobelium in downtime -- this is a temporary test host for discovery and is not ops actionable
* 13:47 Krenair: revert commit for https://gerrit.wikimedia.org/r/#/c/249971/ on tin
* 13:47 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.4/extensions/VisualEditor/VisualEditor.hooks.php: https://gerrit.wikimedia.org/r/#/c/249999/ (duration: 00m 17s)
* 11:57 paravoid: upgrading tor to latest stable on radium
* 11:31 paravoid: ignoring asw-d-eqiad on librenms
* 11:25 _joe_: powercycling dataset1001, stuck in some kernel task, unable to login from console
* 10:42 dcausse: elastic in eqiad setting indexing trace threshold to -1 for commons, ruwiki, frwiki and itwiki
* 10:30 dcausse: elastic in eqiad wide settings for indexing slow threshold are ineffective, trying to set index settings (T117181)
* 09:18 dcausse: elastic in eqiad setting index.indexing.slowlog.threshold.index.trace to -1
* 08:46 _joe_: removed production-search-eqiad_index_indexing_slowlog.log.{7,8} on es1008
* 06:24 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.4/includes/jobqueue/JobQueueRedis.php: (no message) (duration: 00m 18s)
* 05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 30 05:11:08 UTC 2015 (duration 11m 7s)
* 04:19 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d0337b58: ScribuntoSlowFunctionThreshold: 0.90 -> 0.99 (duration: 00m 17s)
* 04:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/common/Hooks.php: I60b9eb617: When logging perf stats, include wfWikiId() in metric key (duration: 00m 18s)
* 04:03 awight: update CRM from e0cad7215192464941d0ec282a70cb608619cf2f to 60c4a8e5fc4efb6ac38a520e652afe187e8c177b
* 03:42 awight|parentoid: update CRM from 367cd85954697c6ad4311deb2799fe9ba08b5d9d to e0cad7215192464941d0ec282a70cb608619cf2f
* 02:47 awight|parentoid: donations queue consumer reenabled
* 02:45 awight|parentoid: updated CRM from 61c8c13efb31f7e75564d9a01fa879db0690fb78 to 367cd85954697c6ad4311deb2799fe9ba08b5d9d
* 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-30 02:30:50+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 13s)
* 00:48 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialUserlogin.php: T117027 (duration: 00m 18s)
* 00:34 mutante: labvirt1010 - started salt-minion (no such process)
* 00:34 mutante: mw1193 - restarted HHVM (socket timeout)
* 00:34 mutante: wdqs1001 - restarted NTP (unknown offset)
* 00:09 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow/: Fix regressions from SWAT (duration: 00m 20s)
* 00:08 mutante: radium - back in service
* 00:01 bd808: Restarted logstash on logstash1003 again. The first try apparently didn't take
 
== 2015-10-29 ==
* 23:44 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibac0d60bd: Disable ScribuntoGatherFunctionStats (duration: 00m 17s)
* 23:32 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: https://gerrit.wikimedia.org/r/249922 (duration: 00m 17s)
* 23:32 bd808: Restarted logstash on logstash1003; died with OOM error
* 23:24 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/engines/LuaSandbox/Engine.php: I69e9218 (duration: 00m 18s)
* 23:21 mutante: radium: scheduled downtime, reinstalling
* 23:19 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/: Fix unwritable cluster errors (duration: 00m 19s)
* 23:18 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Flow: Fix CAPTCHA rendering in RTL languages (duration: 00m 19s)
* 23:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/extensions/Scribunto/: Make the percentile threshold for slow function stats configurable (duration: 00m 18s)
* 23:16 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.4/resources/src/mediawiki.special/mediawiki.special.search.css: SWAT: styling tweaks for inline interwiki search (duration: 00m 18s)
* 23:09 awight: set dashboard cache interval to 60m
* 22:33 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/extensions/ZeroBanner/includes/ZeroSpecialPage.php: T116821 (duration: 00m 17s)
* 22:31 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066, fixes some exceptions in production (duration: 00m 17s)
* 22:30 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/specials/SpecialSearch.php: c249899, fixes some warnings in production (duration: 00m 17s)
* 22:29 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/Category.php: c249890, fixes some warnings in production (duration: 00m 18s)
* 22:05 awight: CiviCRM upgrade to 4.6.9 succeeded; new data is backed up
* 22:02 gwicke: restbase: switched  local_group_default_T_parsoid_html to Date-Tiered compaction (DTCS)
* 21:40 logmsgbot: tgr@tin Synchronized php-1.27.0-wmf.4/includes/MagicWord.php: T117066 (duration: 00m 18s)
* 21:32 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/includes/libs/Xhprof.php: (no message) (duration: 00m 18s)
* 21:06 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Cite/Cite_body.php: Revert "Avoid counting arrays if not needed" (duration: 00m 17s)
* 20:56 gwicke: restbase: switch local_group_wikiquote_T_title__revisions to Date-Tiered compaction (DTCS)
* 20:47 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 19:50 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I06879b6e6e: Enable ScribuntoGatherFunctionStats (duration: 00m 17s)
* 19:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.3
* 19:39 twentyafterfour: rolling back to 1.27.0-wmf.3 due to increase in log errors
* 19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.4
* 18:59 cmjohnson1: powering down aqs1001 for h/w maintenance
* 17:53 AaronSchulz: Touched/synced PrivateSettings.php symlink via touch -h
* 17:47 logmsgbot: aaron@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 18s)
* 17:36 AaronSchulz: Did touch of InitialiseSettings.php
* 17:33 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
* 17:32 logmsgbot: aaron@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 17s)
* 17:17 logmsgbot: kartik@tin Synchronized wmf-config/PrivateSettings.php: Really sync right file this time (duration: 00m 17s)
* 17:10 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Retry syncing for CX token (duration: 00m 17s)
* 17:04 awight: restarting CiviCRM schema migration: v4.3.alpha1 -> v4.6.9 -- Estimated completion 21:30 UTC
* 17:02 awight: update crm from 23d2020448f343c8c1b2f4d779be66f57552f935 to 61c8c13efb31f7e75564d9a01fa879db0690fb78
* 16:57 YuviPanda: restart etherpad on etherpad1001
* 16:22 moritzm: removed obsolete mysql 5.5 packages on mw102[2-9], mw1032, mw1053, mw1114, mw1163
* 16:19 _joe_: moved purge_abusefilter from terbium to mw1152
* 15:48 _joe_: moving purge_checkuser from terbium to mw1152
* 15:43 _joe_: moving purge_securepoll from terbium to mw1152
* 15:15 moritzm: removed openjdk-7 on restbase100[1-9] (it's using openjdk-8 for a while)
* 13:17 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Fix ContentTranslationCXServerAuth for CX (duration: 00m 18s)
* 13:14 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: Fix name of JWT token (duration: 00m 18s)
* 12:33 logmsgbot: kartik@tin Synchronized wmf-config/CommonSettings.php: Set ContentTranslationCXServerAuth for CX (duration: 00m 17s)
* 11:24 mobrovac: restbase rolling-restart after config change https://gerrit.wikimedia.org/r/249465
* 10:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 at 100% load. Reduce db1035 weight. (duration: 00m 17s)
* 10:34 _joe_: migrated jobqueue_stats_reporter to mw1152
* 10:21 hashar: restarting Jenkins (java upgrade)
* 10:20 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1065 after maintenance (duration: 00m 18s)
* 10:14 hashar: Upgrading java on gallium and restarting Jenkins
* 10:04 moritzm: removed openjdk-7 from cassandra test hosts (now using openjdk-8)
* 10:01 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable nearby on wikidata (duration: 00m 18s)
* 10:00 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Config for fetching labels on mobile wikidata (duration: 00m 18s)
* 09:51 jynus: shutdown mw1083 to avoid querying the mysql servers with an outdated config/spaming the error logs
* 09:13 jynus: restarting db1065 for regular maintenace
* 08:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1065 for maintenance (duration: 00m 17s)
* 07:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 29 07:04:04 UTC 2015 (duration 4m 3s)
* 06:43 YuviPanda: disabled two fr-tech related catchpoint tests per awight's request
* 06:30 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: 29ccbd248 (duration: 00m 17s)
* 05:46 ebernhardson: finished manually running 3M enwiki/enqueue, enwiki/wikibase-addUsagesForPage, and wikidatawiki/cirrusSearchLinksUpdatePrioritized jobs from mw1011 and mw1012
* 05:38 ebernhardson: restarting elasticsearch on nobelium to attempt to clear up extra log GC pauses in the old generation (50s+)
* 05:37 ebernhardson: restarting elasticsearch on nobelium
* 03:55 gwicke: cassandra *staging*: testing DateTieredCompactionStrategy (https://labs.spotify.com/2014/12/18/date-tiered-compaction/) on wikipedia html and data-parsoid tables
* 03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-29 03:05:52+00:00
* 03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 03s)
* 02:59 YuviPanda: start pybal on lvs1007
* 02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-29 02:46:27+00:00
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 47s)
* 02:05 AaronSchulz: Restarted stuck hhvm on mw1016; dump at /tmp/hhvm.25097.bt
* 01:06 gwicke: lowered gc_grace on wikipedia parsoid html and data-parsoid keyspaces to 24 hours
 
== 2015-10-28 ==
* 23:31 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/Translate/tag/PageTranslationHooks.php: I0e5f2d3b2 (duration: 00m 18s)
* 23:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/249642 (duration: 00m 17s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 17s)
* 23:22 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: (no message) (duration: 00m 17s)
* 23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
* 23:19 logmsgbot: ebernhardson@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 18s)
* 23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/249603 (duration: 00m 17s)
* 23:15 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.4/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/249585 (duration: 00m 19s)
* 23:08 logmsgbot: ebernhardson@tin Synchronized portals: Switch www portals to be deployed from Git, but not being served from anywhere yet (duration: 00m 18s)
* 23:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I0e5f2d3b2 (duration: 00m 18s)
* 22:57 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.4/extensions/CirrusSearch/includes/Hooks.php: I0e5f2d3b2 (duration: 00m 18s)
* 22:38 urandom: Cassandra cleanup on restbase-test2001-a
* 22:31 awight: Fundraising CRM db migration started, estimated completion 10:15 UTC (tomorrow)
* 22:11 awight: update fundraising CRM to civi-4.6.9-deployment branch, from f2fa7b942625b34ede520e11f20e7e0835ecb17d to 23d2020448f343c8c1b2f4d779be66f57552f935
* 21:11 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.4/includes/changes/EnhancedChangesList.php: Fix diff/history links not showing up for ungrouped enhanced RC - https://gerrit.wikimedia.org/r/#/c/249556/ (duration: 00m 19s)
* 21:08 YuviPanda: enable puppet on labstore1001
* 20:55 YuviPanda: reverted changes to nsswitch.conf from puppet run manually on labstore2001
* 20:29 YuviPanda: disable puppet on labstore1001
* 20:20 awight: Disabled fundraising CiviCRM
* 19:52 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.4
* 18:50 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiversity
* 18:42 logmsgbot: legoktm@tin Synchronized wmf-config/mc.php: https://gerrit.wikimedia.org/r/#/c/249463/ (duration: 00m 17s)
* 18:41 ostriches: rolled scap back to master@62a250a, needs puppet changes before new code goes live
* 18:34 gwicke: restbase deploy done
* 18:34 yurik: updated kartotherian & tilerator services
* 18:23 gwicke: rolling deploy of restbase-deploy 3b1f6488f2 to restbase cluster
* 18:18 gwicke: canary deploy of restbase deploy 3b1f6488f2 to restbase1001
* 18:13 Krinkle: mwscript deleteEqualMessages.php --wiki hiwiki
* 18:12 Krinkle: mwscript deleteEqualMessages.php --wiki fawiki
* 17:59 godog: rolling-restart cassandra-metrics-collector, staggered this time
* 17:23 ostriches: deployed scap master@f823129 to cluster
* 17:16 ostriches: deployed scap master@abe1973 to cluster
* 16:54 godog: unblacklist 'max' cassandra metrics and restart cassandra-metrics-collector
* 16:54 moritzm: installed openjdk security updates on zookeeper hosts
* 16:18 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: add settings for displaying labels on test.wikidata in mobile (duration: 00m 18s)
* 16:15 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: enable wikibase descriptions on test.wikidata (duration: 00m 17s)
* 15:46 andrewbogott: testing the bot after the nfs move
* 15:39 logmsgbot: legoktm@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.search.js: https://gerrit.wikimedia.org/r/#/c/249405/ (duration: 00m 18s)
* 14:22 kart_: T112626 Finished running fix-stats.php for CX (from rwwiki to zuwiki)
* 14:21 moritzm: repooled restbase1006
* 14:09 moritzm: depooled restbase1006 for kernel update
* 14:05 moritzm: repooled restbase1005
* 13:54 moritzm: depooled restbase1005 for kernel update
* 13:47 moritzm: repooled restbase1004
* 13:35 moritzm: depooled restbase1004 for kernel update
* 13:34 moritzm: repooled restbase1003
* 13:16 moritzm: depooled restbase1003 for kernel update
* 13:12 moritzm: repooled restbase1002
* 12:56 moritzm: depooled restbase1002 for kernel update
* 11:46 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 00m 19s)
* 10:39 moritzm: updated kernel on restbase1001 to latest 3.19
* 10:37 _joe_: manually removed crontab from mw1152, erroneously created by puppet
* 10:11 _joe_: reimaging mw1152
* 09:15 _joe_: preparing to reimage mw1152, disabling puppet, scheduling downtime.
* 06:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 28 06:22:06 UTC 2015 (duration 22m 5s)
* 05:15 legoktm: ran mwscript updateSpecialPages.php --wiki=testwiki --only=GadgetUsage on terbium
* 03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.4) at 2015-10-28 03:03:34+00:00
* 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 06m 00s)
* 02:39 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-28 02:39:23+00:00
* 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 33s)
* 00:55 yurik: deployed graphoid - https://gerrit.wikimedia.org/r/#/c/249324/
* 00:37 logmsgbot: tgr@tin Finished scap: Updating MediaViewer with r246112 (duration: 43m 18s)
* 00:35 mutante: mw1135 temp. unresponsive - OOM killer killing hhvm
* 00:27 mutante: powercycling unresponsive mw1127
 
== 2015-10-27 ==
* 23:53 logmsgbot: tgr@tin Started scap: Updating MediaViewer with r246112
* 23:13 ori: running sync-common on mw2050, mw2128, mw2187 and mw2209 (cf I324134438955c7)
* 23:12 gwicke: ran `sudo mdadm --readwrite md1` on restbase1007 to resolve `pending` state
* 23:11 gwicke: rebooted restbase1007 to rule out a funky hardware state causing elevated read latencies
* 23:06 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
* 23:05 logmsgbot: krenair@tin Synchronized wikiversions-labs.json: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 17s)
* 23:05 logmsgbot: krenair@tin Synchronized dblists/all-labs.dblist: https://gerrit.wikimedia.org/r/#/c/248639/ (duration: 00m 18s)
* 23:01 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/249306, no-op (duration: 00m 18s)
* 22:39 mutante: powercycling unresponsive analytics1039, here's what i saw on mgmt https://phabricator.wikimedia.org/P2248
* 21:50 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.4/includes/MovePage.php: 7a7c7b27d6c (duration: 00m 17s)
* 21:50 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Fix for cache key bug (duration: 00m 20s)
* 21:29 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.4
* 21:29 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/debug/logger/LoggerFactory.php: I437bcb532: LoggerFactory: Only check for Psr\Log\LoggerInterface once (duration: 00m 18s)
* 19:46 logmsgbot: twentyafterfour@tin Finished scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch (duration: 31m 32s)
* 19:44 bd808: Ran sync-common on mw2187 to rebuild l10n caches
* 19:38 ejegg: enabled donation queue consumer
* 19:15 logmsgbot: twentyafterfour@tin Started scap: sync everything for 1.27.0-wmf.4 and point testwiki to the new branch
* 19:09 awight: updated DjangoBannerStats from 57a0392b3f43b65050b01a0465e120ed609a769e to 12e819a04a40ee6fab5dd55fcaf072661df31106
* 17:56 paravoid: updating jessie debian-installer to 20150422+deb8u2
* 17:50 Jeff_Green: add fundraising-banner-logger hosts to icinga/nsca
* 17:40 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I95db9deefe363a65 (duration: 00m 17s)
* 17:37 ejegg: disabled donation queue consumer to build up queue for benchmark
* 17:34 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki.ui: I54c195541: Get rid of CSS transitions on form elements in mediawiki.ui (duration: 00m 17s)
* 16:07 Jeff_Green: round of fundraising OS updates, occasional icinga noise is expected
* 16:02 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.BookletLayout.js: SWAT: mw.ForeignStructuredUpload: Mark description as being in source wikis content language [[gerrit:249081]] (duration: 00m 17s)
* 15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor in the "Projet" namespace on the French Wikipedia [[gerrit:248910]] (duration: 00m 17s)
* 15:15 godog: reenable puppet on graphite1001
* 15:01 kart_: TT112626 Ran fix-stats.php for CX (from bewiki to ruwiki)
* 13:00 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/249107/ (duration: 00m 18s)
* 12:59 akosiaris: disabling puppet and bringing down OTRS service on mendelevium
* 12:21 jynus: Just dropped msg_resource tables from labs dbs. Filters modified to stop replicationg them. Started replicating the heartbeat tables.
* 11:56 godog: reimage restbase-test2003
* 11:23 godog: cassandra OOM'd on restbase1007, restarting
* 10:57 godog: downtime restbase endpoints health for restbase1* while investigating
* 10:56 hashar: Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-doc/ is broken. I am on it :-(
* 10:50 hashar: stopping Jenkins due to an unclean state
* 10:11 jynus: disabling puppet and restarting mysql servers at db1069- this will create a small amount of lag on labs
* 09:55 godog: convert restbase-test2003 to cassandra multi-instance
* 09:08 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add forceParse UpdaterFlag and option in forceSearchIndex script (duration: 00m 19s)
* 05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 27 05:47:02 UTC 2015 (duration 47m 1s)
* 03:42 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/languages/Language.php: hotfix for T116693 (duration: 00m 19s)
* 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-27 02:29:08+00:00
* 02:24 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 06s)
* 01:02 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: T116693 (duration: 00m 19s)
* 00:42 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: I2f84cff0: Avoid pointless range scan for 'load-recent-authors' (T116557) (duration: 00m 18s)
* 00:11 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Parsoid/Utils.php: https://gerrit.wikimedia.org/r/#/c/249026 (duration: 00m 18s)
 
== 2015-10-26 ==
* 23:29 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248371/ (duration: 00m 20s)
* 23:22 logmsgbot: krenair@tin Synchronized wmf-config/extension-list-labs: https://gerrit.wikimedia.org/r/#/c/248632/ (duration: 00m 17s)
* 23:19 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/vecwiki.png: https://gerrit.wikimedia.org/r/#/c/248633/ (duration: 00m 18s)
* 23:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248478/ (duration: 00m 17s)
* 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
* 23:15 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/248475/ (duration: 00m 17s)
* 23:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/248871/3 (duration: 00m 18s)
* 22:39 ejegg: updated payments from 7fd1e880b6a45d79fc305b39f3fee2f324324136 to 4baa5a66a4510414d6b43b59f1b1cda2341c17fd
* 21:48 cmjohnson1: powering off wdqs1001 to update idrac settings
* 21:33 ejegg: update payments from 71d2d927f4efac1a639aaf7627c765f48d1b129c to 7fd1e880b6a45d79fc305b39f3fee2f324324136
* 20:50 subbu: deployed parsoid version 660c59a9
* 20:21 ebernhardson: started copy of eqiad elasticsearch indices to noeblium
* 20:19 YuviPanda: stress test on nobelium complete, CPU temperature didn't go above 65C
* 20:13 paravoid: deactivating ulsfo<->NTT BGP peering due to upcoming network migration
* 20:05 YuviPanda: running stress on nobelium
* 19:36 cmjohnson1: swapped bad disk on db1030
* 15:44 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 18s)
* 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 17s)
* 15:31 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Undeploy eventlogging search schema from CirrusSearch (duration: 00m 18s)
* 15:21 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s)
* 15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s)
* 15:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/: Move search schema from cirrussearch -> wikimediavents (duration: 00m 17s)
* 15:06 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Update satisfaction schema id due to bad varnish caching of old id (duration: 00m 17s)
* 15:04 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Remove Predundant Page and Index namespaces from $wgContentNamespaces (duration: 00m 17s)
* 14:58 bblack: repooling cp1059 varnish mobile frontend (wiped)
* 14:18 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on Wikidata (duration: 00m 17s)
* 14:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable geosearch on test.wikidata (duration: 00m 17s)
* 13:32 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (updated submodule) (duration: 00m 18s)
* 13:27 logmsgbot: aude@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (duration: 00m 18s)
* 09:42 dcausse: deleting unused elasticsearch indices in eqiad (T112863)
* 09:20 _joe_: restarting etcd on conf1001
* 09:16 jynus: rebooting and installing jessie on db2060-db2070
* 05:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 26 05:48:19 UTC 2015 (duration 48m 18s)
* 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-26 02:31:46+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
 
== 2015-10-25 ==
* 05:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 25 05:36:33 UTC 2015 (duration 36m 32s)
* 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-25 02:28:08+00:00
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 59s)
 
== 2015-10-24 ==
* 19:34 twentyafterfour: deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium
* 06:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s)
* 05:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s)
* 05:11 hoo: Set an email address for user "Ymnes", after request. Confirmed by several, including.
* 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00
* 02:33 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s)
* 00:21 ebernhardson: started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads
 
== 2015-10-23 ==
* 23:31 Krinkle: mwscript deleteEqualMessages.php --wiki hewikisource
* 23:31 Krinkle: mwscript deleteEqualMessages.php --wiki ukwiki
* 23:09 hoo: Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer
* 22:23 robh: powercycled nobelium
* 21:24 mutante: nobelium - powercycled, no console output
* 20:04 jynus: rename user job had created lag on almost all enwiki dbs, things should be better now
* 19:08 urandom: starting nodetool cleanup on restbase-test2001-b
* 19:07 urandom: starting nodetool cleanup on restbase-test2001-a
* 18:16 godog: bounce grafana-server on krypton
* 17:58 YuviPanda: moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch
* 17:57 godog: delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries
* 17:53 YuviPanda: stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856
* 17:13 akosiaris: enable_notifications=0 in neon's icinga for a few mins while the storm dies down
* 17:04 cwd: updated payments from 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c to 71d2d927f4efac1a639aaf7627c765f48d1b129c
* 16:47 godog: stop puppet on graphite1001 and graphite-index cron, suspected root cause
* 16:26 godog: uwsgi timing out while serving requests, bounce also carbon daemons
* 16:20 godog: bounce uwsgi for graphite-web on graphite1001
* 16:14 urandom: starting rebuild of restbase-test2002-b
* 15:42 jynus: copied wmf-mariadb10 (10.0.16-2) .deb from trusty to jessie on apt.wikimedia.org
* 15:42 urandom: bouncing restbase-test2002-a Just To See
* 15:06 urandom: running nodetool cleanup on restbase-test2003
* 14:47 godog: remove outdated cassandra metrics from graphite2001
* 14:27 godog: remove  restbase-test2001 restbase-test2002 cassandra metrics
* 14:12 mobrovac: restbase rolling-restarting after applying aab840f
* 13:55 jynus: Rebooting and installing jessie on db2055-db2059
* 13:45 godog: remove 98percentile 999percentile meanRate stddev cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
* 13:34 godog: remove 15MinuteRate cassandra metrics after https://gerrit.wikimedia.org/r/#/c/248313/
* 11:51 godog: roll-restart cassandra-metrics-collector on restbase cluster after  https://gerrit.wikimedia.org/r/248313
* 10:27 jynus: retrying schema change on EventLogging (db1046) after failure
* 10:21 jynus: End of online schema change to geo_tags; all wikis on dblist have been updated
* 09:34 godog: reimage restbase-test2002.codfw.wmnet
* 09:23 jynus: restarting schema change on geo_tags for all wikis
* 09:17 mobrovac: restbase rolling-restart after config changes
* 06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 18s)
* 06:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilterTokenizer.php: I65d4c6064: Track tokenizer cache hits / misses (duration: 00m 17s)
* 05:53 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 23 05:53:43 UTC 2015 (duration 53m 42s)
* 03:15 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/profiler/TransactionProfiler.php: 5ef4a91480ea (duration: 00m 18s)
* 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-23 02:41:25+00:00
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 04s)
* 00:10 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/anwiki.png: https://gerrit.wikimedia.org/r/#/c/247253/ (duration: 00m 16s)
 
== 2015-10-22 ==
* 23:58 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/247850/ (duration: 00m 17s)
* 23:49 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: https://gerrit.wikimedia.org/r/#/c/248273/ (duration: 00m 18s)
* 23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
* 23:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/autoload.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
* 23:12 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/SpamFilter/RateLimits.php: https://gerrit.wikimedia.org/r/#/c/248229/ (duration: 00m 17s)
* 22:55 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: extensions/Cite update which fell off the SWAT train yesterday (duration: 00m 19s)
* 22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
* 22:02 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaMaintenance/getJobQueueLengths.php: Ie95ec067da9: getJobQueueLengths: add '--report' option for StatsD reporting (duration: 00m 18s)
* 21:11 logmsgbot: reedy@tin Synchronized docroot and w: Add more dblist symlinks (duration: 00m 18s)
* 21:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/changetags/ChangeTags.php: e7126ed331109 (duration: 00m 17s)
* 21:05 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes: a6262272c9666d (duration: 00m 23s)
* 20:39 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner/: Deploying ZeroBanner T116309 patch 248239 (duration: 00m 18s)
* 20:38 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/MobileFrontend: Deploying MobileFrontend T116309 patch 248238 (duration: 00m 36s)
* 20:26 bd808: Removed "zirconium.wikimedia.org" from Trebuchet's minion list for iegreview/iegreview
* 20:12 bd808: Updated iegreview.wikimedia.org to bcaf23b (Fix logger usage in Controllers\Account\Recover)
* 19:53 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.3/extensions/ZeroBanner: Deploying ZeroBanner T116309 patch 248116 (duration: 00m 18s)
* 19:28 bd808: Forced ELK Elasticsearch to allocate replica of logstash-2015.10.22 shard 0 on logstash1004
* 18:56 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.27.0-wmf.3
* 16:48 bd808: Restarted Elasticsearch on logstash1004
* 16:39 logmsgbot: kartik@tin Synchronized private/PrivateSettings.php: T116134: Set CX JWT token (duration: 00m 17s)
* 16:19 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8f690589: Increase abusefilter emergency disable threshold on MediaWiki.org (duration: 00m 17s)
* 15:30 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf/highlight.php: SWAT: Remove urlencode from phabricator links [[gerrit:248049]] (duration: 00m 17s)
* 15:13 logmsgbot: thcipriani@tin Synchronized docroot/noc/conf: SWAT: noc: change Gitblit links to Diffusion [[gerrit:248027]] (duration: 00m 17s)
* 15:11 moritzm: uploaded openjdk-8 8u66-b17 for jessie-wikimedia to carbon
* 14:46 jynus: performing schema change on eventlogging database on db1046
* 14:41 mutante: mw1083 - Error: Could not run Puppet configuration client: Read-only file system
* 14:25 jynus: setting thread_pool_size to 32 dynamically on all MariaDB hosts
* 14:01 jynus: performing schema change on officewiki-flow (s3)
* 13:19 jynus: performing schema change on x1-master (flowdb)
* 12:31 jynus: Rolling schema change for GeoData on all wikis (geo_tags)
* 11:35 mobrovac: restbase deployed 2bc05f40
* 07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 00m 17s)
* 07:41 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/jobqueue/JobQueueRedis.php: Ie7c544fc8: jobqueue: track real job inserts as inserts_actual & I627e8f6ce: JobQueueRedis::doBatchPush(): report metrics even when failures occur (duration: 07m 33s)
* 06:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 22 06:17:39 UTC 2015 (duration 17m 38s)
* 06:14 AaronSchulz: Restarted hhvm on mw1011, it was stuck doing nothing at 100 cpu
* 05:10 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksUpdate.php: fe323f9b68bbb (duration: 00m 17s)
* 03:05 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-22 03:05:21+00:00
* 03:00 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 07m 52s)
* 02:37 AaronSchulz: Started running 5 threads of enwiki refreshLinks jobs on tin
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-22 02:35:41+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 44s)
* 01:10 logmsgbot: krenair@mira Synchronized README: testing sync from mira (duration: 00m 17s)
* 01:01 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
* 01:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/Hooks.php: I0e5f2d3b2: Make hookErrorHandler() only care about serious signature errors (duration: 00m 17s)
* 00:56 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
* 00:55 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 17s)
* 00:49 logmsgbot: krenair@mira Synchronized README: (no message) (duration: 00m 16s)
* 00:20 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ibf752b832: $wgMathCheckFiles = false (duration: 00m 18s)
* 00:05 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
* 00:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter: Ice1b6da43: AbuseFilter: don't install custom error handler and I0ecdcdd142: Use isset() to check array element exists rather than relying on @ operator (duration: 00m 18s)
 
== 2015-10-21 ==
* 23:33 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 19s)
* 23:32 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Performance tweaks for corss-dc copy process (duration: 00m 18s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Provide category suggestions from the right wiki (duration: 00m 17s)
* 23:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/resources/: mw.ForeignStructuredUpload: Rearrange messages to always display license name (duration: 00m 18s)
* 23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throtle exception for eswiki (duration: 00m 18s)
* 23:03 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
* 23:00 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/extensions/AbuseFilter/AbuseFilter.parser.php: Ad-hoc debug logging of AbuseFilter exceptions (duration: 00m 17s)
* 21:10 ebernhardson: starting copy of elasticsearch eqiad indices to codfw
* 20:46 ebernhardson|lch: cancel copying elasticsearch eqiad to labsearch, looks to be writing to wrong disks and will fill up
* 20:16 AaronSchulz: Started running 8 threads of commonswiki refreshlinks jobs on terbium
* 20:14 ebernhardson|lch: starting copy of elasticsearch indices from eqiad cluster to labsearch cluster
* 18:42 ebernhardson: initializing elasticsearch index mapping for all wikis in the codfw and labsearch ES clusters
* 18:12 ejegg|mtg: updated crm from 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435 to f2fa7b942625b34ede520e11f20e7e0835ecb17d
* 16:53 hoo: Attached WeiaR@enwiki to the global account of the same name. T115699
* 15:50 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
* 15:49 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Revert SearchSatisfaction schema related changes due to suspected perf impact (duration: 00m 18s)
* 15:38 akosiaris: depooled mw1083
* 15:29 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: touch and re-sync InitialiseSettings.php to bust cache (duration: 00m 17s)
* 15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:24 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch multi-datacenter configuration (duration: 00m 17s)
* 15:18 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 17s)
* 15:16 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Cite/Cite_body.php: Do not double-parse error references duplicate key (duration: 00m 19s)
* 15:00 aude: rsync failed on mw1083: "failed to set times on "/srv/mediawiki/wmf-config": Read-only file system"
* 14:49 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on test.wikidata (duration: 00m 18s)
* 14:34 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings-labs.php: enabled geodata on beta wikidata (duration: 00m 18s)
* 14:08 jynus: applying schema change to testwikidatawiki (s3) and wikidatawiki(s5)
* 13:56 jynus: performing schema change on testwiki.geo_tags
* 13:35 jynus: stopping and fixing replication on labsdb1004 (not in production)
* 08:40 mobrovac: restbase deployment of 3006b77e {{done}}
* 08:32 mobrovac: restbase deploying 3006b77e
* 07:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 21 07:36:04 UTC 2015 (duration 36m 3s)
* 03:13 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-21 03:13:42+00:00
* 03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 24s)
* 02:42 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-21 02:42:44+00:00
* 02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 05s)
* 01:12 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
* 01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: revert (duration: 00m 18s)
* 01:03 logmsgbot: hoo@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 17s)
* 00:29 logmsgbot: legoktm wikidata'd everything
* 00:22 logmsgbot: hoo@tin Synchronized dblists/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 17s)
* 00:19 logmsgbot: hoo@tin Synchronized wmf-config/: Enable WikibaseClient on mediawikiwiki, metawiki and specieswiki (duration: 00m 19s)
 
== 2015-10-20 ==
* 23:54 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (duration: 00m 18s)
* 23:47 logmsgbot: hoo@tin Synchronized wmf-config/: Add MediaWiki, Meta-Wiki and Wikispecies to Wikibase special site groups (testwikidata) (duration: 00m 18s)
* 23:36 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEventsHooks.php: https://gerrit.wikimedia.org/r/#/c/247650/ (duration: 00m 17s)
* 23:26 logmsgbot: hoo@tin Synchronized wmf-config/: Revert "Temporarily disable "item-merge" right on Wikidata" (duration: 00m 18s)
* 23:25 logmsgbot: aaron@tin Synchronized php-1.27.0-wmf.3/includes/deferred: 2a1e1d7dd88a62aba9 (duration: 00m 17s)
* 23:22 logmsgbot: hoo@tin Synchronized wmf-config/: Bump the cache epoch for (test)wikidata (duration: 00m 18s)
* 23:20 logmsgbot: hoo@tin Finished scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies (duration: 48m 44s)
* 23:16 mutante: mw1232: restarted hhvm
* 22:31 logmsgbot: hoo@tin Started scap: Update Wikibase to wmf3b and add messages for sitelinks to MediaWiki, Meta-Wiki and Wikispecies
* 21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 17s)
* 21:11 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.2/includes/page/WikiPage.php: I5d0440588d: Make triggerOpportunisticLinksUpdate() directly use RefreshLinks (T116001) (duration: 00m 18s)
* 20:23 gwicke: reverted restbase deploy on restbase1001 to a4c55e40
* 20:15 mutante: starting restbase on restbase1001
* 20:09 mutante: errors argon' on argon
* 20:04 ottomata: uninstalling hadoop packages on analytics1017
* 19:46 ori: previous syncs was of I46200d4edb3: Revert "Revert "Revert "Enable config for all three search clusters, but only write to eqiad"""
* 19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 18s)
* 19:45 logmsgbot: ebernhardson@tin Synchronized wmf-config/: (no message) (duration: 00m 18s)
* 18:25 mutante: re-enabling icinga notifications for icinga (neon) itself that were disabled for some reason though all OK
* 16:50 mutante: temp. disabled ircecho / neon puppet
* 16:50 jynus: enabling query profiling on a sample of queries on db1072
* 16:27 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Revert Temporarily increase redis logging to debug (duration: 00m 18s)
* 16:25 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Temporarily increase redis logging to debug (duration: 00m 17s)
* 16:21 aude: re-populated sites table on metawiki, mediawikiwiki and specieswiki with https protocol links
* 16:13 cwdent: updated fundraising crm from 738e8c3f8079765841ed4c5f79ecf066c541c7b9 to 22dc4bd7d041126a1d2a0d4acb9a288bfdc1b435
* 15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part V (duration: 00m 18s)
* 15:48 godog: bump netdev_max_backlog to 10000 on graphite1001, T101141
* 15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part IV (duration: 00m 17s)
* 15:46 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part III (duration: 00m 17s)
* 15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part II (duration: 00m 17s)
* 15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Revert "Revert "Enable config for all three search clusters, but only write to eqiad"" [[gerrit:247478]] Part I (duration: 00m 17s)
* 15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation suggestion in all Wikipedia [[gerrit:247515]] (duration: 00m 17s)
* 15:23 awight: update fundraising CRM from 6ad0ad090c23ee41003138a6676131abf70c72f4 to 738e8c3f8079765841ed4c5f79ecf066c541c7b9
* 15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch graphoid to the local restbase proxy [[gerrit:247494]] (duration: 00m 17s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Move ForeignUploadTargets config to production [[gerrit:246703]] part 2 (duration: 00m 17s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Move ForeignUploadTargets config to production [[gerrit:246703]] part I (duration: 00m 18s)
* 14:33 cmjohnson1: removing tele2(patchid 2953) from dmarc panel @eqiad
* 14:12 ottomata: deployed varnishreqstats diamond collector to remaining varnish caches
* 13:48 jynus: backing up and renaming user_daily_contribs table from all wikis as a previous step for its deletion
* 12:44 kart_: Update cxserver to 6452b68
* 12:01 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Temporarily disallow item-merge until T115892 is resolved (duration: 00m 19s)
* 11:16 mobrovac: mathoid deploying 8e1a3327
* 07:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 20 07:23:56 UTC 2015 (duration 23m 55s)
* 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-20 03:14:15+00:00
* 03:09 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 16s)
* 02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Bring phase0 and phase1 inline with phase2 (duration: 00m 18s)
* 02:48 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Elastica/: Bring phase0 and phase1 inline with phase2 (duration: 00m 21s)
* 02:44 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-20 02:43:53+00:00
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 20s)
 
== 2015-10-19 ==
* 23:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: testwiki Graphoid to restbase (duration: 00m 17s)
* 23:13 logmsgbot: ebernhardson@tin Synchronized wmf-config/: resync after touching InitialiseSettings.php to bust caches (duration: 00m 18s)
* 23:12 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Revert multidc cirrussearch config, seeing unexplained errors on commonswiki (duration: 00m 18s)
* 23:11 mutante: restarted hhvm on mw1231
* 23:09 logmsgbot: ebernhardson@tin Synchronized wmf-config/throttle.php: Add throttle exception for dewiki (duration: 00m 17s)
* 23:08 mutante: restarted apache on mw1231
* 23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Disable cirrus suggester AB test (duration: 00m 17s)
* 23:05 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Bump sampling rate of common terms test from 1:1000 to 1:200 (duration: 00m 17s)
* 23:01 logmsgbot: ebernhardson@tin Synchronized tests/: noop sync mediawiki-config test dir (duration: 00m 17s)
* 22:42 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable cirrusearch multi cluster configuration, only write to eqiad (duration: 00m 18s)
* 22:40 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Handle ElasticaWrite job failures internally (duration: 00m 18s)
* 22:29 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Deploy multi-dc cirrusearch code for CirrusSearch extension (duration: 00m 18s)
* 22:28 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Elastica/: Deploy multi-dc cirrusearch code for Elastica extension (duration: 00m 17s)
* 22:20 ebernhardson: sync-common on mw1017 to pre-test cirrussearch multi-dc deployment
* 22:17 paravoid: salt-run deluser --delete-home gmetric; delgroup systemusers
* 20.23 subbu: cherrypick deploy for parsoid completed: b317f33f and 60a82ae0 cherrypicked from parsoid master
* 15:42 logmsgbot: anomie@tin Started scap: SWAT: Add a change tag to cross-wiki uploads [[gerrit:246701]]
* 15:39 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref [[gerrit:247255]] (duration: 00m 18s)
* 15:30 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Cite: SWAT: Display 'cite_error_references_duplicate_key' next to the affected ref [[gerrit:247256]] (duration: 00m 18s)
* 15:25 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob [[gerrit:247268]] (duration: 00m 18s)
* 15:24 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/includes/deferred/LinksDeletionUpdate.php: SWAT: Use specified pageId for LinksDeletionUpdate→DeleteLinksJob [[gerrit:247267]] (duration: 00m 17s)
* 15:12 urandom: deploying a4c55e40 to RESTBase
* 15:11 joal: Scheduling icinga downtime for CQL checks on aqs while heavily loading data - joal (me) babysites the jobs - 1 day downtime, will reiterate tomorrow if needed
* 15:09 ottomata: enabling varnish reqstats diamond collector on all upload caches
* 15:09 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Search/Connection.php: SWAT: Backport [[gerrit:246134]] because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
* 15:07 logmsgbot: anomie@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/includes/Search/Connection.php: SWAT: Backport [[gerrit:246134]] because the thing it fixed suddenly started breaking unit tests, preventing other merges (duration: 00m 18s)
* 14:52 akosiaris: disabled puppet on maps-test200{1,2,4}. Debugging cassandra multi-instance setup aftermath. Not to be enabled
* 14:34 urandom: canary deploy (a4c55e40) to restbase1001.eqiad
* 12:47 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl-eng_0.1.0~r20599-1
* 12:22 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-isl_0.1.0-1
* 06:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 19 06:19:09 UTC 2015 (duration 19m 8s)
* 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-19 03:04:49+00:00
* 02:59 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 40s)
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-19 02:36:53+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 15s)
 
== 2015-10-18 ==
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-mlt-ara_0.1.0~r57554-1
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-is-sv_0.1.0~r56030-1
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-ro_0.7.3~r57551-1
* 10:04 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-es-it_0.1.0~r51165-1
* 06:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 18 06:11:37 UTC 2015 (duration 11m 36s)
* 02:57 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-18 02:57:48+00:00
* 02:52 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
* 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-18 02:29:50+00:00
* 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 25s)
 
== 2015-10-17 ==
* 20:02 godog: powercycle analytics1034, no console no ssh
* 14:05 godog: reboot krypton, unable to ssh and no console (VM) iowait through the roof
* 06:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 17 06:27:04 UTC 2015 (duration 27m 3s)
* 03:03 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-17 03:02:58+00:00
* 02:58 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s)
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-17 02:35:14+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 09s)
 
== 2015-10-16 ==
* 20:52 hashar: Restarting Jenkins to remove potential dead locks before the week-end
* 20:48 ejegg: updated payments from 33b3bd6bee11b3cc9de1570584a23354d0b6525f to 4d38158f3d3a3a6da85d809a6cdc557e46c45d0c
* 19:29 ori: deleted /var/lib/carbon/whisper/MediaWiki/MediaWiki on graphite1001 & graphite2001 per tgr's request
* 16:14 awight: Update paymentswiki logging config per T107918
* 15:52 awight: Fundraising: remove FR fraud exception
* 13:18 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I14ecd0ae87: Turn off UserDailyContribs extension (duration: 00m 18s)
* 06:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 16 06:25:56 UTC 2015 (duration 25m 55s)
* 03:11 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-16 03:11:14+00:00
* 03:06 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 31s)
* 02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-16 02:43:42+00:00
* 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 08m 38s)
* 02:27 jynus: repairing table and restarting replication on s7 from labsdb1002
* 02:23 jynus: repairing table and restarting replication on s5 from dbstore1002 (non-production host)
* 01:36 Jamesofur: reset email address for User:INeverCry after identify verification
 
== 2015-10-15 ==
* 22:09 greg-g: don't worry, _joe_ was around and we approved Roan's last deploy as an exception :)
* 22:08 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.3/extensions/Flow/: Back out categories in sidebar feature (duration: 00m 20s)
* 20:30 _joe_: rebooting ms-be1011
* 19:41 awight: update crm from ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1 to a4e74f6f38e6cff16ecf79d28d6e4de9499a5017
* 19:08 cwdent: updated payments from bdec3220030a396e2a447763e40b940a332e2ab8 to 33b3bd6bee11b3cc9de1570584a23354d0b6525f
* 18:41 ejegg: updated refund queue name in IPN listener settings
* 18:13 ejegg: updated CiviCRM from 17dc351d92a8437c93b4a9fa2385840b3581dad6 to ddfaa209f5b5af4aa0cf3403da91d39b3c52acc1
* 17:18 cwdent: updated payments from ba8f80ec7a858074cd3856a52f3758cf96571f67 to bdec3220030a396e2a447763e40b940a332e2ab8
* 09:54 _joe_: restarted gitblit, unresponsive
* 05:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 15 05:41:25 UTC 2015 (duration 41m 24s)
* 02:51 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-15 02:51:27+00:00
* 02:49 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 05m 02s)
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-15 02:35:15+00:00
* 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 57s)
 
== 2015-10-14 ==
* 23:46 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Nevermind on skipping timed text, extension config is weird (duration: 00m 17s)
* 22:26 akosiaris: restarted zotero on sca1002
* 21:54 akosiaris: restarted zotero on sca1001. complaining out of memory in logs
* 20:46 ejegg: updated civicrm from ba39f3181431c8409416e580dcf6e15ac5f96a21 to 17dc351d92a8437c93b4a9fa2385840b3581dad6
* 20:20 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Also exclude timedtext ns from purges (duration: 00m 18s)
* 19:52 robh: reenabled puppet agent on etherpad1001, was disabled for a few hours and no reason specified and no SAL entry
* 19:50 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn off the mw "warning" logging channel (duration: 00m 18s)
* 19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Tighten check on empty page purges (duration: 00m 18s)
* 19:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I1d7969a9d: Purge pages with blank content beyond the PP limit report, I38cba9d9b6a: Get strpos() parameters correct (duration: 00m 17s)
* 19:10 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I3a8a750bb3: Enable T115505 log channel for I1d7969a (duration: 00m 17s)
* 18:06 logmsgbot: demon@tin Synchronized wmf-config: (no message) (duration: 00m 19s)
* 17:59 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
* 17:58 logmsgbot: demon@tin Synchronized wmf-config/extension-list: (no message) (duration: 00m 17s)
* 15:31 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: SWAT: Split connection to source and target [[gerrit:246243]] (duration: 00m 18s)
* 15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files [[gerrit:246225]] (duration: 00m 17s)
* 15:14 ori: restarted navtiming and statsv services on hafnium
* 15:11 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.3/extensions/UploadWizard/UploadWizard.config.php: SWAT: Remove default category for UploadWizard files [[gerrit:246226]] (duration: 00m 18s)
* 15:05 Krinkle: Statsv and eventlogging-navtiming seems to have gone down 7 hours ago
* 14:24 ori: cleaned up tessera leftovers on graphite1001
* 14:15 mutante|1way: mw1157 - deleted puppet lock file, fix puppet run. ("already running" but didnt since 18h)
* 13:30 ottomata: kafka-preferred-replica election after kafka1012's broker restarted last night
* 13:12 ottomata: restarted diamond, puppet didn't seem to after it removed the TcpConnStates from most hosts
* 12:55 hashar: ERROR: Could not connect to SMTP host: polonium.wikimedia.org, port: 25  (from labs instances)
* 12:32 logmsgbot: krenair@tin Synchronized README: testing https://gerrit.wikimedia.org/r/246206 (duration: 00m 17s)
* 12:07 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.3/extensions/VisualEditor/modules/ve-mw/ui/inspectors/ve.ui.MWLinkAnnotationInspector.js: https://gerrit.wikimedia.org/r/#/c/246205/ (duration: 01m 13s)
* 03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 14 03:14:22 UTC 2015 (duration 14m 21s)
* 03:14 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-14 03:14:22+00:00
* 03:07 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 10m 37s)
* 02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-14 02:40:43+00:00
* 02:37 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 10s)
* 00:19 ejegg: updated civicrm from c7af7634e75eb8702f5e16081a86ab86ce69c7c2 to ba39f3181431c8409416e580dcf6e15ac5f96a21
 
== 2015-10-13 ==
* 23:30 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default (duration: 01m 14s)
* 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/WikimediaEvents/: Turn on cirrus common terms test (duration: 01m 15s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/CirrusSearch/: Add information for common terms ab test (duration: 01m 15s)
* 23:21 Krenair: restarted apache on silver
* 23:19 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.2/extensions/Flow: Make flow board descriptions editable again (duration: 01m 16s)
* 23:13 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/Flow: Bump flow submodule in 1.27.0-wmf.3 (duration: 01m 15s)
* 23:10 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/: Update WME for common terms AB test (duration: 01m 14s)
* 23:08 logmsgbot: ebernhardson@tin Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch/: Update cirrus for common terms AB tes (duration: 01m 15s)
* 23:02 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Turn on logging of the "warning" channel (duration: 01m 13s)
* 23:00 ejegg: updated SmashPig from b5ff2a7d5f17aaaa33a169ca101cbea639769c90 to c431c8d77521270236c72532a50806b2e852cf7b
* 21:05 ottomata: reenabling puppet on cp1052
* 20:51 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
* 20:48 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/includes/db: I0e5f2d3b2: Revert Enforce lagged-slave read-only mode on the DB layer (duration: 01m 14s)
* 20:45 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.3/autoload.php: (no message) (duration: 01m 14s)
* 20:40 akosiaris: restart gitblit (once more)
* 20:27 _joe_: rebooting mw1157, stuck in a kernel soft lockup
* 20:22 ori: locally hacked testwiki and mediawikiwiki to point to php-1.27.0-wmf.3 on mw1017
* 20:19 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
* 20:18 logmsgbot: ori@tin rebuilt wikiversions.cdb and synchronized wikiversions files: (no message)
* 20:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.3
* 19:59 ejegg: updated civicrm f25fbe856b92373104985185db77311ea3a4d841 to c7af7634e75eb8702f5e16081a86ab86ce69c7c2
* 19:38 JohnFLewis: delete wikidata-l mailing list (archives still accessible)
* 19:30 logmsgbot: twentyafterfour@tin Finished scap: Full scap sync for 1.27.0-wmf.3 (duration: 31m 26s)
* 18:58 logmsgbot: twentyafterfour@tin Started scap: Full scap sync for 1.27.0-wmf.3
* 18:45 ottomata: FYI i have puppet disabled on cp1052 while I try to figure out a diamond+VSL+multiprocessing bug
* 18:11 mutante|1way: started salt on mw2083
* 18:09 mutante|1way: restarted gitblit
* 17:58 ori: testing ldap integration in grafana 2
* 17:54 urandom: performing deploy of a4c55e4 to restbase staging
* 17:42 urandom: performing canary deploy of a4c55e4 to restbase staging (xenon)
* 16:14 ostriches: created education program tables for srwiki, T110619
* 16:14 mobrovac: restbase deploying a01c62a6
* 16:08 ottomata: restarting diamond on cp1052 in gdb in attempt to figure out why vanrishreqstats segfaults...
* 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions for de, fa, fi, he, nn, pa, pl and te wikipedias [[gerrit:245862]] (duration: 01m 13s)
* 15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default" (duration: 01m 13s)
* 15:15 ori: grafana-test: Imported Grafana dashboards from ElasticSearch
* 14:12 bblack: note to self: we should migrate all our service to Java
* 14:11 bblack: restarted gitblit on antimony
* 13:16 bblack: repooling cp2017 (codfw upload)
* 02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 13 02:32:17 UTC 2015 (duration 32m 16s)
* 02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-13 02:32:17+00:00
* 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 07m 01s)
* 01:53 logmsgbot: faidon@tin Synchronized wmf-config/CommonSettings.php: unbreak BounceHandler (duration: 01m 14s)
* 00:51 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Reapply: Flow-occupy talk namespaces on sewikimedia (duration: 01m 13s)
* 00:29 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/Flow/modules/flow-initialize.js: https://gerrit.wikimedia.org/r/#/c/245596/ (duration: 01m 13s)
* 00:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244724/ (duration: 01m 13s)
* 00:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245608/ (duration: 01m 13s)
* 00:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 01m 13s)
* 00:00 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245588/ and https://gerrit.wikimedia.org/r/#/c/245589/ (duration: 01m 13s)
 
== 2015-10-12 ==
* 23:53 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/245578/ (duration: 01m 12s)
* 23:38 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243920/ (duration: 01m 14s)
* 23:35 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/243921/ (duration: 01m 13s)
* 23:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244378/ (duration: 01m 13s)
* 23:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244896/ (duration: 01m 14s)
* 23:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245194/ (duration: 01m 13s)
* 23:11 logmsgbot: krenair@tin Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/243517/ (duration: 01m 13s)
* 23:06 logmsgbot: krenair@tin Synchronized database lists: (no message) (duration: 01m 13s)
* 20:21 bearND: MobileApps deployed sha1 95293e5
* 19:09 Tim: on ruthenium installed iotop for stall investigation
* 16:13 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/ZeroBanner: SWAT: Defer loading of ZeroOverlay until needed [[gerrit:244737]] (duration: 01m 13s)
* 16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Ada Lovelace Day editathon 2015 [[gerrit:245472]] (duration: 01m 13s)
* 15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:ShortURL on bnwiki [[gerrit:244953]] (duration: 01m 14s)
* 15:39 logmsgbot: thcipriani@tin Synchronized dblists/commonsuploads.dblist: SWAT: Remove duplicate entries from commsuploads.dblist [[gerrit:244435]] (duration: 01m 12s)
* 15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use new page name for wmf release notes [[gerrit:241079]] (duration: 01m 14s)
* 15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Azerbaijani Wikisource project and namespaces [[gerrit:242096]] (duration: 01m 13s)
* 15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add GeoData and PageImages configuration for Wikibase repo wikis [[gerrit:244165]] (duration: 01m 13s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Explicitly set wmgMFNearby = false for wikidata [[gerrit:244591]] (duration: 01m 14s)
* 15:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Modify timezone for cswiktionary [[gerrit:244649]] (duration: 01m 12s)
* 14:54 dcausse: closing unused cirrus indices in eqiad (T112863)
* 14:28 cmjohnson: rebooting mw1154
* 12:10 yurik: deployed kartotherian & tilerator to maps-test200{1-4}
* 08:54 hashar: zuul-merger process leaked file descriptors and ended up unable to open any more files.  Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243
* 08:44 hashar: Zuul CI in trouble.  zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243
* 02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s)
* 02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s)
 
== 2015-10-11 ==
* 04:57 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 11 04:57:40 UTC 2015 (duration 57m 39s)
* 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-11 02:28:34+00:00
* 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 17s)
 
== 2015-10-10 ==
* 06:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 10 06:40:04 UTC 2015 (duration 40m 3s)
* 02:41 ejegg|afk: enabled fundraising banner campaigns
* 02:36 ejegg|afk: updated payments-wiki from 24d5be6886d34b3600031290c7f55ee84f3dcee2 to ba8f80ec7a858074cd3856a52f3758cf96571f67
* 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-10 02:22:30+00:00
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 58s)
 
== 2015-10-09 ==
* 22:20 ejegg: took down Fundraising campaigns
* 19:33 logmsgbot: ori@tin Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s)
* 18:13 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s)
* 16:04 godog: rolling restart cassandra test cluster T95253
* 15:35 urandom: performing Cassandra cleanup on restbase-test2003.codfw
* 15:15 urandom: bouncing Cassandra on restbase-test2001-a
* 15:07 urandom: starting nodetool cleanup on restbase-test2002
* 14:37 jynus: more configuration testing (with puppet disabled) and several mysql restarts on db1022
* 13:53 hashar: Restarted Zuul, had a deadlocked job
* 11:50 godog: bounce mathoid on sca100[12], stray instance found not running firejail
* 11:22 mobrovac: restbase deploying aaee7c31
* 11:16 godog: force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/
* 11:00 jynus: restarting db1022's mysql (depooled) for configuration testing
* 09:17 jynus: deployed visual glitch fix to dbtree
* 09:11 akosiaris: poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
* 09:11 akosiaris: poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing.
* 08:20 ori: Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/
* 05:13 mutante: @RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s)
* 01:24 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s)
* 01:18 RoanKattouw: Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh
* 01:17 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s)
 
== 2015-10-08 ==
* 23:56 logmsgbot: catrope@tin Finished scap: SWAT (duration: 29m 10s)
* 23:51 yurik: reverted Kartotherian to HEAD^^ - the service wouldn't start
* 23:27 logmsgbot: catrope@tin Started scap: SWAT
* 23:06 yurik: deployed kartotherian
* 18:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2
* 16:45 mobrovac: mathoid deploying 110abaf
* 15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) [[gerrit:244332]] (duration: 00m 18s)
* 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki [[gerrit:244142]] (duration: 00m 17s)
* 14:53 paravoid: replacing cr1-codfw<->asw-a-codfw QSFPs
* 14:48 logmsgbot: reedy@tin Synchronized docroot and w: (no message) (duration: 00m 17s)
* 14:13 jynus: performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002)
* 13:59 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s)
* 13:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s)
* 12:09 godog: update facts on puppet compiler from palladium
* 11:38 _joe_: rebooting mc2002
* 10:39 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s)
* 10:37 _joe_: rebooting mw1008, in oom spiral
* 10:15 paravoid: salt rm /home/*/.ssh/authorized_keys
* 09:21 jynus: downtime for db1055 for maintenance (kernel update, mysql update, config update)
* 08:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 17s)
* 07:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase load of db1051 and db1055 also for regular traffic (duration: 00m 17s)
* 07:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Matching db1051 and db1055 weight for load balancing (duration: 00m 16s)
* 06:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct  8 06:15:16 UTC 2015 (duration 15m 15s)
* 05:56 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 18s)
* 03:04 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-08 03:04:52+00:00
* 03:02 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 33s)
* 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-08 02:46:57+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 30s)
* 01:17 twentyafterfour: phabricator is now up and running again
* 01:09 twentyafterfour: buggy update, stopping apache2 on iridium.
* 01:02 twentyafterfour: finished phabricator update
* 00:51 mutante: mw1160 - was oom-killer (convert, mw job 31497)
* 00:50 twentyafterfour: phabricator maintenance/upgrade. Expect 10 minutes downtime
* 00:44 mutante: powercycled mw1160 - looks like broken hardware or cable (ata1: lost interrupt)
* 00:08 logmsgbot: krenair@tin Synchronized w/static/apple-touch/commons.png: https://gerrit.wikimedia.org/r/#/c/243685/ (duration: 00m 17s)
 
== 2015-10-07 ==
* 23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242842/ (duration: 00m 17s)
* 23:34 logmsgbot: ori@tin Synchronized wmf-config: I924d8e19e17: Make the redis cache configuration multi-DC-ready (T111575) (duration: 00m 17s)
* 23:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244177/ (duration: 00m 17s)
* 23:18 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244176/ (duration: 00m 17s)
* 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243837/ (duration: 00m 17s)
* 21:25 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Revert "Don't use nutcracker on wikitech" (duration: 00m 16s)
* 20:56 mutante: applied firewalling on IRCd server, rc bot still working fine, all public IRC ports as before
* 20:25 mutante: argon: installing package upgrades
* 20:24 mutante: resetting drac on argon
* 20:14 logmsgbot: ori@tin Synchronized wmf-config/session.php: Ie25c368a: Switch mw1017 to use DC-specific redis cluster names (duration: 00m 17s)
* 19:50 urandom: RESTBase deploy complete
* 19:41 urandom: doing full deploy of c20e6336 to RESTBase
* 19:29 urandom: canary deploy to restbase1001.eqiad complete
* 19:14 urandom: deploying c20e6336 to canary node restbase1001.eqiad
* 19:09 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.2
* 18:56 logmsgbot: twentyafterfour@tin Synchronized wmf-config/CommonSettings.php: fix undefined variable warning that has been spamming logs (duration: 00m 17s)
* 17:56 awight_: update fundraising crm from 7003cc38797848631d0c4d5f6ff68ab1d6118ad8 to f25fbe856b92373104985185db77311ea3a4d841
* 16:39 mutante: powercycling analytics1035
* 16:14 mobrovac: citoid deploying ec149fd5
* 16:02 logmsgbot: marktraceur@tin Synchronized wmf-config/: Adding new config variable for uploads to Commons (duration: 00m 17s)
* 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/MobileFrontend: SWAT (duration: 00m 17s)
* 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 18s)
* 15:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
* 15:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1051 after maintenance (duration: 00m 17s)
* 15:03 logmsgbot: krenair@tin Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 17s)
* 15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242041/ (duration: 00m 18s)
* 14:59 mobrovac: AQS restarting restbase on aqs100x
* 13:00 godog: decomission restbase-test2001 and reimage
* 12:27 paravoid: salt-rm'ing /var/lib/apt/lists/ubuntu.wikimedia.org_ubuntu_dists_trusty_main_i18n_Translation-en%5fUS
* 10:55 godog: reenable puppet on restbase / maps-test / aqs
* 10:10 _joe_: depooling cp1059 from pybal, varnish
* 08:44 godog: disable puppet on restbase, maps, aqs before merging https://gerrit.wikimedia.org/r/#/c/243127
* 08:05 moritzm: installed spice security updates on labvirt*
* 06:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct  7 06:01:01 UTC 2015 (duration 1m 0s)
* 05:25 ebernhardson: generating elasticsearch indices in codfw, should run ~3 hours
* 04:33 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: Touch InitiialiSettings.php to force config regeneration (duration: 00m 18s)
* 04:23 logmsgbot: ebernhardson@tin Synchronized wmf-config/: enable second es cluster in testwiki one more time (duration: 00m 18s)
* 04:15 ebernhardson: sync-common on mw2187.codfw.wmnet to fix localisation cache errors in exception.log
* 03:37 logmsgbot: ebernhardson@tin Synchronized wmf-config: redisable second cluster only on testwiki (duration: 00m 16s)
* 03:35 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Reenable second ES cluster on testwiki only (duration: 00m 18s)
* 03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-07 03:10:02+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 10m 18s)
* 02:37 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-07 02:37:32+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
* 01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: Fix JS error (duration: 00m 18s)
* 01:44 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: Fix JS error (duration: 00m 17s)
* 00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.2/extensions/Echo: SWAT (duration: 00m 17s)
* 00:04 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo: SWAT (duration: 00m 17s)
 
== 2015-10-06 ==
* 23:20 logmsgbot: ebernhardson@tin Synchronized wmf-config: Revert multicluster config for testwiki (duration: 00m 18s)
* 23:18 logmsgbot: ebernhardson@tin Synchronized wmf-config/: Enable multicluster ES on testwiki (duration: 00m 17s)
* 22:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
* 22:40 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.2: Deploy https://gerrit.wikimedia.org/r/#/c/244066/ (duration: 01m 40s)
* 21:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
* 21:15 twentyafterfour: restarted phd in response to a phabricator setup issue
* 20:58 bblack: disabling puppet on caches for VCL testing
* 20:58 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.2
* 20:53 logmsgbot: twentyafterfour@tin Finished scap: sync wmf/1.27.0-wmf.1 (duration: 32m 13s)
* 20:29 mutante: service "ishmael" has been removed (T109777) - removed docroot on neon. tarball exists in /root just in case. code is on https://github.com/asher/ishmael
* 20:20 logmsgbot: twentyafterfour@tin Started scap: sync wmf/1.27.0-wmf.1
* 18:48 yuvipanda: fixing https://phabricator.wikimedia.org/T109216 on labstore1002
* 18:10 godog: upgrade videoscalers to ffmpeg2theora 0.29.0~git+20150813-2
* 15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Suggestion in af, gl, gu, mk, oc, sh and simplewiki [[gerrit:243919]] (duration: 00m 18s)
* 11:33 jynus: performing schema change on db1051 enwiki revision
* 11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1051 for more maintenance (duration: 00m 17s)
* 10:45 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1051 after maintenance (duration: 00m 17s)
* 10:41 jynus: potential extra load on mediawiki recent changes and watchlist on enwiki, please report any slowdown
* 10:22 jynus: dropping temp recovered tables from db1051 to prepare for repool
* 09:50 _joe_: uploaded a new pybal package for jessie
* 06:19 paravoid: eqord is back up
* 04:24 paravoid: all waves to eqord down, probably related to RT#9619
* 02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-06 02:34:31+00:00
* 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 53s)
 
== 2015-10-05 ==
* 23:53 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/243833/ (duration: 00m 17s)
* 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: rv (duration: 00m 17s)
* 23:14 logmsgbot: krenair@tin Synchronized php-1.27.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui: https://gerrit.wikimedia.org/r/#/c/243729/ (duration: 00m 17s)
* 23:05 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/242942/ (duration: 00m 17s)
* 21:16 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Take2: Deploying ZeroBanner 242661+243808 (duration: 00m 17s)
* 21:00 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Rolling back ZeroBanner 242661 (duration: 00m 18s)
* 20:55 logmsgbot: yurik@tin Synchronized php-1.27.0-wmf.1/extensions/ZeroBanner: Deploying ZeroBanner 242661 (duration: 00m 17s)
* 18:38 mutante: restarted gitblit
* 17:44 gwicke: rolling restart of eqiad restbase nodes done
* 17:27 gwicke: rolling restart of restbase cluster to rule out driver issues causing the increased p99 read latency
* 17:19 robh: analytics1035 pegged out, ssh unresponsive and raid failures, and then fixed itself 5 minutes later
* 16:32 legoktm: running backPopulateRenameQueueLogs.php ([[gerrit:237169]]) on metawiki
* 16:05 hoo: Updated Wikidata's property suggester with data from today's json dump
* 15:19 logmsgbot: thcipriani@tin Synchronized php-1.27.0-wmf.1/extensions/Flow: SWAT: Fix exception on board and topic history pages [[gerrit:243337]] (duration: 00m 20s)
* 14:41 hashar: puppet-lint  Jenkins job is now strict and will -1 on errors as well as warnings https://gerrit.wikimedia.org/r/#/c/243185/
* 10:25 godog: roll-restart cassandra in restbase eqiad
* 09:58 _joe_: installing the new HHVM package to appservers
* 09:50 _joe_: rebooted mw1153, soft lockup due to bnx2 failure
* 09:50 godog: roll-restart cassandra in restbase codfw
* 09:18 godog: roll-restart cassandra on restbase test cluster
* 09:15 godog: stop puppet on restbase and maps in preparation for https://gerrit.wikimedia.org/r/#/c/242896/1
* 08:07 _joe_: upgrading HHVM on all API appservers
* 05:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct  5 05:33:18 UTC 2015 (duration 33m 17s)
* 02:33 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-05 02:33:57+00:00
* 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 25s)
 
== 2015-10-04 ==
* 23:36 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/extension.json: 8c80ec1273: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation (duration: 00m 17s)
* 05:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct  4 05:13:32 UTC 2015 (duration 13m 31s)
* 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-04 02:31:56+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 06s)
 
== 2015-10-03 ==
* 15:33 jynus: stopping temporarily labsdb1004 mariadb to complete clone process
* 09:38 _joe_: rolling restarting all parsoids in eqiad
* 09:14 _joe_: restarting parsoid on wtp1021
* 05:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct  3 05:28:54 UTC 2015 (duration 28m 53s)
* 03:12 ori: done; graphite-web back up; url shortening will now work.
* 03:11 ori: shutting down graphite-web for brief sqlite database schema update
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-03 02:36:33+00:00
* 02:31 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 22s)
 
== 2015-10-02 ==
* 22:46 mutante: holmium: apt-get clean for a little disk space - /var/log/designate/designate-mdns.log is more than half the size of / - needs logrotate
* 21:33 Krinkle: mwscript cleanupRemovedModules.php --wiki zhwiktionary
* 21:21 Krinkle: mwscript cleanupRemovedModules.php --wiki testwikidatawiki
* 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki dewiki
* 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki nlwiki
* 21:15 Krinkle: mwscript cleanupRemovedModules.php --wiki test2wiki
* 21:06 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 17s)
* 18:39 ottomata: kafka preferred-replica-election
* 17:53 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I680f3fda66c5: Configure ResourceLoader-specific ObjectCache instance (duration: 00m 17s)
* 17:53 ottomata: rolling restart of all hadoop-yarn-nodemanagers to pick up python3 + spark fix
* 17:12 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I8318fe892: Configure MultiWriteBagOStuff for ResourceLoader (duration: 00m 17s)
* 16:29 ori: deployed grafana 8e92884bae (with backport of upstream fb9f9548829f2d4cecf35cda933700e5c2fa1bd6)
* 15:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/resources/src/mediawiki/mediawiki.js: I8029208: Don't clobber existing styles when adding more in IE9 (duration: 00m 17s)
* 14:37 jynus: providing extra grants to wiki db users for heartbeat monitoring
* 14:32 Coren: Change of NFS mount options in labs pushed - puppet may report failures to refresh the mounts (once) on instances; expected and harmless.
* 11:03 moritzm: installed rpcbind security updates on all Ubuntu servers which runs it (jessie was already updated, since the DSA was released earlier)
* 10:53 jynus: restarting HHVM on mw1130
* 08:45 hashar: restarting Nodepool to take in account changes made to the logging configuration https://gerrit.wikimedia.org/r/#/c/240986/
* 06:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct  2 06:07:05 UTC 2015 (duration 7m 4s)
* 02:42 logmsgbot: tstarling@tin Synchronized php-1.27.0-wmf.1/extensions/ParsoidBatchAPI: stats (duration: 00m 17s)
* 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-02 02:41:33+00:00
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 08m 04s)
* 00:44 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: I21bb3f08e7f and follow-ups (duration: 00m 18s)
* 00:43 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/vendor: Add wikimedia/relpath 1.0.3 (duration: 00m 21s)
 
== 2015-10-01 ==
* 22:04 RoanKattouw: Running FlowFixLinks.php on all wikis
* 20:33 subbu: deployed parsoid version 62971510b
* 19:33 twentyafterfour: re-deployed robots.txt patch and restarted apache on iridium (to expand the phabricator robots.txt)
* 18:22 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader: Ic1d802ee2: ResourceLoader: cache minified user and site modules (duration: 00m 17s)
* 18:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.1
* 16:40 ottomata: brought analytics1049 back into hadoop after missing a disk since sept. 25th
* 16:34 andrewbogott: test log
* 16:26 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
* 16:26 valhallasw`cloud: testing, testing
* 15:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/240049 (duration: 00m 17s)
* 15:56 cmjohnson1: db1051 replacing failed disk slot 6
* 15:51 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/extensions/ContentTranslation/modules/dashboard/styles/ext.cx.dashboard.less: Fix: Clicking on down arrow in language selector should trigger ULS (duration: 00m 17s)
* 15:48 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Enable CX suggestions in ar, eo, hi, nl, vi and dawiki (duration: 00m 17s)
* 15:47 cmjohnson1: analytics1049 replacing failed disk /dev/sdi at slot 7
* 15:46 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 18s)
* 15:45 logmsgbot: mattflaschen@tin Synchronized php-1.27.0-wmf.1/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 17s)
* 15:44 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/tests/phpunit/includes/objectcache/BagOStuffTest.php: Memcached key decode fix (duration: 00m 17s)
* 15:43 logmsgbot: mattflaschen@tin Synchronized php-1.26wmf24/includes/objectcache/MemcachedBagOStuff.php: Memcached key decode fix (duration: 00m 18s)
* 15:33 cmjohnson1: replacing failed disk ms-be1012 /dev/sdf slot 5
* 15:09 matt_flaschen: Did final run of convertAllLqtPages.php on sewikimedia immediately before freezing LQT
* 15:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings.php: Freeze LQT on se.wikimedia (duration: 00m 18s)
* 14:58 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
* 14:19 andrewbogott: log test
* 14:17 andrewbogott: testing the log
* 14:07 _joe_: installing the new hhvm package to all the canaries
* 13:41 hashar: Am I logging?
* 13:32 _joe_: uploaded hhvm_3.6.5+dfsg1-1+wm7
* 13:03 bblack: repooling cp1046 (eqiad mobile) with caches wiped clean just before
* 12:24 akosiaris: disable SessionRemoteIPcheck on mendelevium's OTRS installation for checking
* 12:18 logmsgbot: aude@tin Finished scap: Put Wikidata extension back on wmf/1.27.0-wmf.1 (duration: 30m 46s)
* 12:09 moritzm: installed PHP security updates on all precise/trusty systems (the respective DSA for jessie is already deployed, it was released three weeks ago)
* 11:47 logmsgbot: aude@tin Started scap: Put Wikidata extension back on wmf/1.27.0-wmf.1
* 10:35 logmsgbot: hoo@tin Synchronized php-1.27.0-wmf.1/extensions/: Use 1.27.0-wmf.1 for Wikidata again after fixing T114290 (duration: 01m 07s)
* 10:20 akosiaris: uploaded php5_5.3.10-1ubuntu3.20+wmf1 on apt.wikimedia.org precise-wikimedia
* 09:51 godog: reboot praseodymium to test cassandra systemd unit
* 09:47 godog: stop puppet on restbase* in preparation for    https://gerrit.wikimedia.org/r/242548
* 08:59 akosiaris: disabling puppet on maps-test200* in expectance of https://gerrit.wikimedia.org/r/242548
* 06:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct  1 06:56:51 UTC 2015 (duration 56m 50s)
* 03:21 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-01 03:21:39+00:00
* 03:14 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 38s)
* 02:47 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-10-01 02:47:31+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 54s)
* 01:30 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
* 01:29 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoader.php: 1cfe27030e: Change load.php to minify per-module instead of per-request (duration: 00m 17s)
* 00:44 mutante: applying puppet fix on fermium (illegal byte sequence in utf-8) as in T114289#1691038 but for other languages
 
== 2015-09-30 ==
* 23:00 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
* 22:55 logmsgbot: twentyafterfour@tin Synchronized php-1.27.0-wmf.1/: deploying several fixes to the branch (duration: 01m 52s)
* 22:44 ottomata: perf testing eventlogging in production by hammering https://bits.wikimedia.org/beacon/event.gif
* 22:13 logmsgbot: ori@tin Synchronized php-1.27.0-wmf.1/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
* 22:12 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s)
* 21:36 cwdent: updated payments from bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11 to 24d5be6886d34b3600031290c7f55ee84f3dcee2
* 21:27 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: a1e1619461: Fix LESS file dependency tracking in ResourceLoader (duration: 00m 17s)
* 20:43 cscott: updated Parsoid to version 39c60c67
* 20:40 mutante: fixing puppet run on fermium, needs manual fix because puppet cant replace existing illegal character in some templates
* 19:18 logmsgbot: krinkle@tin Synchronized php-1.27.0-wmf.1/resources/Resources.php: T114288 (duration: 00m 17s)
* 18:51 jynus: stopping replication on s2, s3 and s7 for dbstore1001
* 18:50 ottomata: restarted eventlogging with blacklist=^Analytics$
* 18:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
* 18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1
* 17:39 ottomata: restarting eventlogging with 12 client side processors
* 16:28 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader: 89595ba49a: Cherry-pick I173a9820b and I7c7546ec (duration: 00m 18s)
* 16:20 logmsgbot: krenair@tin Finished scap: swat (duration: 53m 33s)
* 15:27 logmsgbot: krenair@tin Started scap: swat
* 13:53 moritzm: rebooted stat1001/stat1002/stat1003 for kernel updates (already happened between 13:00 UTC and 13:10 UTC, but forgot to log earlier)
* 12:34 moritzm: added debdeploy 0.0.8 to carbon
* 05:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 30 05:43:19 UTC 2015 (duration 43m 18s)
* 03:10 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-09-30 03:10:39+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 45s)
* 02:36 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-30 02:36:37+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 21s)
* 01:02 logmsgbot: ori@tin Synchronized php-1.26wmf24/vendor: 940124a7db: Updated mediawiki/core Project: mediawiki/vendor  ff5e254f7eddf811f6f66b4a4063b1a8cc70f265 (duration: 00m 21s)
 
== 2015-09-29 ==
* 23:59 ottomata: restarting eventlogging so that processors use etcd to pick up shared token with which to consistently hash IPs
* 23:46 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo/: SWAT (duration: 00m 17s)
* 23:37 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on mediawikiwiki (duration: 00m 16s)
* 23:33 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 18s)
* 23:32 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/Echo/: SWAT (duration: 00m 17s)
* 23:30 logmsgbot: catrope@tin Synchronized php-1.27.0-wmf.1/extensions/GuidedTour/GuidedTourHooks.php: T114144 Fix back button logging (duration: 00m 17s)
* 22:50 ejegg: updated payments-wiki from 3b0915a51a0fd567bdf22f3d4e17548a83e735d8 to bc4bcc44d2337d7a69c5a39f11ff45efdf0c8e11
* 22:47 ejegg: updated SmashPig from bf302444eae8236734fd43883b06c7b2512b1532 to 513ec01123e6dbb97b00888a3610a7c5ec24a63b
* 22:38 twentyafterfour: finished deployment of 1.27.0-wmf.1 to group0
* 22:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.1
* 22:34 logmsgbot: twentyafterfour@tin Finished scap: php-1.27.0-wmf.1/extensions/ (duration: 04m 29s)
* 22:30 twentyafterfour: 22:27:17 sync-dir failed: <ValueError> /srv/mediawiki-staging/php-1.27.0-wmf.1/vendor/oyejorge/less.php/lib/Less/Version.php has content before opening <?php tag
* 22:30 chasemp: es-tool unban-node elastic1031
* 22:29 logmsgbot: twentyafterfour@tin Started scap: php-1.27.0-wmf.1/extensions/
* 22:23 chasemp: unbanning elastic1006 for shard population
* 22:05 ori: 22:03:51 Synchronized php-1.26wmf24/extensions/CentralNotice: 30bdfcb386: Updated mediawiki/core Project: mediawiki/extensions/CentralNotice  6bd658e155a02edb4cc506bc3494a3f4699d3e94 (duration: 00m 17s)
* 19:34 logmsgbot: twentyafterfour@tin Finished scap: sync php-1.27.0-wmf.1 for validation on testwiki (duration: 31m 49s)
* 19:30 bd808: Updated iegreview.wikimedia.org to c3ac5e6 (Update to Twig 1.20.0) and applied latest schema changes
* 19:02 logmsgbot: twentyafterfour@tin Started scap: sync php-1.27.0-wmf.1 for validation on testwiki
* 18:11 cmjohnson1: shutting down elastic1031 to relocate rack/row
* 18:09 cmjohnson1: shutting down elastic1006 to relocate row/rack
* 17:46 mutante: size of conntrack table on iron might be increased due to test scans
* 17:32 ejegg: updated appeal template setting on payments-wiki
* 16:30 cmjohnson1: swapping failed disk db1050
* 16:14 subbu: reverted configuration hotfix from yesterday's Parsoid deploy (re-enabled use of Parsoid batching API)
* 15:58 bblack: ending VCL tests on cp1053
* 15:52 bblack: ending VCL tests on cp1065, starting on cp1053 instead
* 15:39 bblack: cp1065: live-testing some VCL patches, puppet disabled, etc...
* 15:19 chasemp: ban elastic1031 for T112559
* 15:05 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: VisualEditor: Set TransitionDefault true for the English Wikipedia [[gerrit:242040]] (duration: 00m 17s)
* 14:08 paravoid: repooling eqiad; 24h codfw test window is over
* 10:49 godog: bounce pybal on lvs2003 / lvs2006
* 09:13 paravoid: powercycling pybal-test2003
* 08:51 jynus: starting cloning of labsdb1005 (Tools DB), minimal disruption is expected
* 07:03 twentyafterfour: restarted apache2 and phd or iridium to get phabricator back into the correct state
* 06:58 twentyafterfour: checked out correct phabricator release tag on iridium. Something, somewhere, had reverted everything to an old deployment.
* 05:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 29 05:01:30 UTC 2015 (duration 1m 29s)
* 02:34 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-29 02:34:55+00:00
* 02:30 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 57s)
* 02:25 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp*, actually
* 02:24 bblack: ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp20*
* 01:55 robh: upload-lb.codfw.wikimedia.org_ipv6 page flap
* 01:43 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Plumbing for wmgVisualEditorTransitionDefault (duration: 00m 17s)
* 01:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add wmgVisualEditorTransitionDefault (false everywhere) (duration: 00m 17s)
* 01:13 robh: didnt powercycle analytics1035 yet, it recovered on its own.
* 01:13 robh: powercycling analytics1035, seems oom, cannot login via ssh or serial console
* 00:56 logmsgbot: tstarling@tin Synchronized php-1.26wmf24/extensions/ParsoidBatchAPI: Fix fatal error I77fd7e8 (duration: 00m 17s)
 
== 2015-09-28 ==
* 23:47 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Update ttmserver configuration to match elasticsearch security profile (duration: 00m 17s)
* 23:43 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I56db35b: Removed ignore_user_abort( true ) line (duration: 00m 18s)
* 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Echo: SWAT (duration: 00m 18s)
* 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow: SWAT (duration: 00m 19s)
* 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/CentralNotice: SWAT (duration: 00m 19s)
* 20:51 bearND: MobileApps deployed sha1  9df72ec
* 20:42 subbu: deployed parsoid version b9e5244e + hotfix on tin to turn off batching api use since canary restart of wtp1002 showed some batching api errors
* 20:26 yuvipanda: depooled cp2017 from pybal config too
* 20:22 yuvipanda: depooled cp2017 since it's down
* 19:02 chasemp: powercycle iridium via console as it's unresponsive
* 18:01 awight: update crm from 190f689ff7aec7fecefdf5af501293685c55e041 to 7003cc38797848631d0c4d5f6ff68ab1d6118ad8
* 15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s)
* 15:28 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s)
* 15:11 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s)
* 14:00 paravoid: running failover eqiad->codfw test for all frontend traffic
* 06:11 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s)
* 05:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s)
* 02:27 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s)
 
== 2015-09-27 ==
* 23:17 logmsgbot: ori@tin Synchronized php-1.26wmf24/./resources/src/mediawiki.toolbar/toolbar.less: I94ced06178: mediawiki.toolbar: temporary workaround for T113868 (duration: 00m 17s)
* 16:51 Krenair: ran sync-common on snapshot1001 to bring it up to date
* 16:43 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf21
* 16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf20
* 16:42 logmsgbot: krenair@tin Purged l10n cache for 1.26wmf19
* 15:50 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/241354/ - fix AbuseFilter block durations (duration: 00m 18s)
* 10:05 jynus: leaving innodb compression tests on es2005 running (could affect lag on that host)
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 05s)
 
== 2015-09-26 ==
* 19:04 hashar: restarting Jenkins. Just in case :-D
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 48s)
 
== 2015-09-25 ==
* 23:54 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s)
* 22:16 mutante: sodium - shutting down
* 21:53 ori: Armed new servicedeploy_rsa on mira
* 21:51 ori: Armed new servicedeploy_rsa on tin
* 20:45 urandom: starting a Cassandra repair on xeon (nodetool repair -pr)
* 19:30 mutante: restarted apache on gallium
* 19:07 bblack: configuring and enabling lvs-cross-row ports on asw-b-eqiad for lvs1007,8,10,11
* 18:04 jynus: performing schema change on m5-master "nova"
* 17:26 jynus: restarting and upgrading db1051 mysql (depooled)
* 17:20 godog: reboot ms-be1005, xfs
* 16:22 cwdent: updated payments from dc78ff5157b59a8f475dc86194a1059c2d6b2fad to 3b0915a51a0fd567bdf22f3d4e17548a83e735d8
* 16:17 akosiaris: duplicating database otrs to otrsupgradetest for testing the upgrade procedure
* 16:16 cmjohnson1: cp1046 stopped icinga checks for hardware troubleshooting
* 14:43 moritzm: installed rpcbind and apport security updates on various servers
* 14:36 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialMovepage.php: https://gerrit.wikimedia.org/r/#/c/241045/ (duration: 00m 17s)
* 14:22 chasemp: 'sudo -u hdfs hdfs haadmin -transitionToActive analytics1001-eqiad-wmnet' per otto on analytics1001
* 14:03 urandom: starting a Cassandra repair on restbase1006 (nodetool repair -pr -dc eqiad)
* 13:58 hashar: nodepool back in operations
* 13:44 moritzm: restarted saltmaster on palladium
* 13:12 mobrovac: restbase deploying e42bf0fc
* 13:12 moritzm: added debdeploy 0.0.7 to carbon
* 13:10 godog: enable puppet on restbase in production
* 12:54 andrewbogott: restarted rabbitmq-server on labcontrol1001
* 12:52 godog: stop puppet on restbase in production -- config deployment
* 12:46 hashar: stopping nodepool to clear out left over mysql connections
* 12:21 hashar: Nodepool is dead, or at least not adding new slaves anymore
* 12:16 hashar: Seems Zuul/Jenkins is in trouble somehow :-/
* 11:42 andrewbogott: upgrading linux-image-generic on labnet1002 to get us away from the recently-crashed 3.13.0-59
* 04:58 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I252970886: Set "async" for SQL parser cache everywhere else (duration: 00m 18s)
* 02:57 Krinkle: sync-common failed on mw1010.eqiad.wmnet
* 02:53 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/DonationInterface: 381faf5 (duration: 00m 21s)
* 02:33 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 11m 45s)
* 01:44 Krinkle: mwscript deleteEqualMessages.php --wiki roa_tarawiki
* 01:42 Krinkle: mwscript deleteEqualMessages.php --wiki itwikiquote
* 01:42 Krinkle: mwscript deleteEqualMessages.php --wiki alswiki
* 01:28 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/WikimediaEvents: I5608f8ffd1c - Fix trailing comma (duration: 00m 17s)
* 00:10 logmsgbot: krinkle@tin Synchronized php-1.26wmf24/extensions/EventLogging/modules/ext.eventLogging.core.js: Increase maxUrlSize to 2000 (duration: 00m 17s)
 
== 2015-09-24 ==
* 23:02 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240880/ (duration: 00m 17s)
* 21:18 ejegg: updated SmashPig from d1baa32267eaad7d69b47c657f4853eb306fad6b to bf302444eae8236734fd43883b06c7b2512b1532
* 21:15 ejegg: updated payments-wiki from 8428499feb8760d63faf681d53995697a2ba0fa7 to dc78ff5157b59a8f475dc86194a1059c2d6b2fad
* 19:57 logmsgbot: ori@tin Synchronized php-1.26wmf24/extensions/ContentTranslation: d079d5dd71: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation  8559ee614975f25b71a732ca0fb1bb6d489c9d33 (duration: 00m 18s)
* 19:35 bblack: depooled cp1046 from confd, committed pybal depool for LVS as well
* 19:34 chasemp: changing labs route on cr1 and cr2 from 10.68.16.0/22 to 10.68.16.0/21 which matches references, fw setting and manifests/network.pp
* 18:54 logmsgbot: catrope@tin Synchronized php-1.26wmf24/extensions/Flow/: Debugging for FlowFixLinks.php (duration: 00m 20s)
* 18:21 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf24
* 18:20 legoktm: moved oauthadmin group from User:Yuvipanda@metawiki to User:YuviPanda@metawiki
* 18:19 godog: restart restbase on restbase1005
* 18:18 godog: restart restbase on restbase1004
* 18:17 ejegg: reenabling CRM jenkins jobs
* 18:16 godog: restart restbase on restbase1003
* 18:08 ejegg: updated civicrm from 9fa38d06a75363a8009bce7ced190e39c75b68bc to 190f689ff7aec7fecefdf5af501293685c55e041
* 18:06 paravoid: depooling cp1046, stability issues
* 18:05 ejegg|afk: disabled CRM jenkins jobs
* 18:00 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 17s)
* 17:59 ori: Merged Apache config change Ia095457fb. It will refresh the Apache service as it rolls out, causing elevated 503s for the next 20 minutes.
* 17:53 godog: rolling restart restbase in eqiad
* 17:35 chasemp: powercycling cp1046 at mgmt as I can't ssh in and it seems like it should be up
* 17:26 godog: bounce restbase on restbase1002, apply new datacenter config
* 17:10 _joe_: cleaning up /tmp on mw1152
* 17:09 cmjohnson1: powering down for the last time es1001 - es1010
* 16:17 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/Wikidata: SWAT: Do not filter affected pages by namespace [[gerrit:240727]] (duration: 00m 26s)
* 16:01 robh: nothing on puppet swat window, easiest swat ever.
* 15:46 logmsgbot: thcipriani@tin Synchronized php-1.26wmf24/extensions/Wikidata: SWAT: Do not filter affected pages by namespace [[gerrit:240711]] (duration: 00m 26s)
* 15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable suggestions in ca, en, es, fr, it, ja, tr, ru, zh [[gerrit:240638]] (duration: 00m 17s)
* 14:37 paravoid: repooling codfw
* 12:54 bblack: restarting varnish daemons on second half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
* 12:50 bblack: restarting varnishd instances on text, mobile, upload clusters for package upgrade (slow salt, no parallelism, ~5m spacing - FE cache loss, BE cache stays, should take ~9h)
* 12:05 moritzm: installed rpcbind security updates on eeden, baham, radon, maerlant, rhenium
* 11:56 bblack: restarting varnish daemons on half of maps, parsoid, misc clusters (package upgrade, shm_reclen change)
* 11:36 bblack: reinstall lvs300[12] to jessie - T96375
* 11:21 akosiaris: killed tail -f varnishncsa.log on cp1065 and ran apt-get clean to reclaim some disk space
* 11:14 bblack: stopping pybal on lvs300[12]; lvs300[34] taking over
* 11:07 bblack: upgrading varnishes to 3.0.6plus-wm8 (non-restarting, just pkg update on-disk)
* 09:40 jynus: performing latest (software) steps to decom es1001-es1010 (puppet disabling, etc.)
* 08:39 jynus: restarted HHVM @ mw1056, 1104, 1122
* 05:33 yuvipanda: deleted logstash indexes for 08/27 and 28 too
* 05:31 yuvipanda: deleted indexes for 08/14, 15, 25, 26 on logstash
* 03:59 yuvipanda: restarting elasticsearch in logstash1001-3
* 03:53 yuvipanda: restarted es on logstash1004-6
* 03:02 yuvipanda: jstack dumped logstash output onto /home/yuvipanda/stack on logstash1001 since strace seems useles
* 02:51 yuvipanda: restarted logstash on logstash1002
* 02:41 yuvipanda: gmond at 100% again, killing it and stopping puppet again
* 02:40 yuvipanda: re-enabling and running puppet on hafnium to see what it's bringing up
* 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
* 02:23 yuvipanda: kill gmond on hafnium and disable puppet to prevent it from taking it back up. Was taking 100% CPU
* 02:16 Krinkle: Kibana/Logstash outage. Zero events received after 2015-09-23T23:59:59.999Z.
* 02:14 Krinkle: Partial EventLogging outage (client-side events via hafnium abruptly stopped 2015-09-23 11:36 UTC - 15 hours ago)
* 01:53 mutante: started logstash on logstash1002 again
* 01:35 mutante: bast1001: unmounting /srv/home_pmtpa (backup on bacula)
* 01:34 mutante: removing subversion packages from bast1001
* 01:15 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes: Ifa0d4cfe8e3: Backport I1ff61153d and I8e4c3d5a5 (duration: 00m 23s)
* 00:19 jynus: restarted replication on db1051
* 00:17 ori: restarting tcpircbot on neon
* 00:16 mutante: started logstash on logstash1002
* 00:08 bblack: varnish package on carbon for jessie updated to 3.0.6plus-wm8
 
== 2015-09-23 ==
* 23:17 logmsgbot: krenair@tin Synchronized php-1.26wmf24/includes/specials/SpecialSearch.php: https://gerrit.wikimedia.org/r/#/c/240596/ (duration: 00m 18s)
* 23:09 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240575/ (duration: 00m 17s)
* 22:53 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 14f46330d9: Backport fix from PS12 of I1ff6115 (duration: 00m 17s)
* 22:39 paravoid: cr1-codfw RE switchover(s)
* 22:35 logmsgbot: yurik@tin Synchronized php-1.26wmf24/extensions/ZeroBanner: Deploying ZeroBanner interstitial handling (duration: 00m 18s)
* 21:13 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 58bfb6f85b: Backport fix from PS9 of I1ff6115 (duration: 00m 17s)
* 20:46 subbu: deployed parsoid 6619409e
* 19:58 urandom: starting Cassandra on restbase2006
* 19:56 urandom: enabling puppet, and forcing a run on restbase2006
* 19:52 urandom: starting Cassandra on restbase2005
* 19:50 urandom: enabling puppet, and forcing a run on restbase2005
* 19:48 urandom: starting Cassandra on restbase2004
* 19:45 urandom: enabling puppet, and forcing a run on restbase2004
* 18:55 twentyafterfour: snapshot1001.eqiad.wmnet returned [12]: rsync: write failed on "/srv/mediawiki/wikiversions.cdb": No space left on device
* 18:54 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24
* 18:01 mobrovac: restbase deploying f65313ed
* 17:49 urandom: starting Cassandra on restbase2003
* 17:45 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 25s)
* 17:35 urandom: enabling, and forcing puppet run on restbase2003
* 17:27 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
* 17:20 logmsgbot: ori@tin scap failed: OSError [Errno 2] No such file or directory: '/var/lock/scap' (duration: 44m 24s)
* 17:11 logmsgbot: ori@tin Synchronized php-1.26wmf24/includes/page/Article.php: (no message) (duration: 00m 13s)
* 17:11 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page/Article.php: (no message) (duration: 00m 17s)
* 16:35 logmsgbot: ori@tin Started scap: (no message)
* 16:35 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/page: I952068d2d: Reduced the DOS potential of 404 page floods (duration: 00m 12s)
* 16:22 godog: start cassandra on restbase2002
* 16:05 godog: deploy restbase daacf4d on restbase2*
* 15:34 logmsgbot: demon@tin Synchronized php-1.26wmf24/extensions/Wikidata: (no message) (duration: 00m 20s)
* 15:34 logmsgbot: demon@tin Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 21s)
* 15:34 chasemp: unbanning elastic1005 for T112559
* 15:32 logmsgbot: demon@tin scap aborted: (no message) (duration: 02m 07s)
* 15:30 logmsgbot: demon@tin Started scap: (no message)
* 15:15 logmsgbot: demon@tin Synchronized php-1.26wmf22/extensions/Wikidata: (no message) (duration: 00m 20s)
* 14:13 logmsgbot: demon@tin Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 12s)
* 14:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
* 14:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 12s)
* 14:04 logmsgbot: demon@tin Synchronized langlist-labs: (no message) (duration: 00m 11s)
* 14:04 logmsgbot: demon@tin Synchronized docroot/noc/conf/: (no message) (duration: 00m 13s)
* 14:01 logmsgbot: demon@tin Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 11s)
* 13:56 godog: force puppet run on restbase2001 to deploy new cassandra config
* 13:28 godog: reboot ms-be1012, xfs hosed
* 13:22 paravoid: upgrading cr1-codfw with newer junos
* 07:43 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: semver the special:version hook (duration: 00m 12s)
* 03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 10m 12s)
* 02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-23 02:38:44+00:00
* 02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 21s)
* 00:56 logmsgbot: legoktm@tin Synchronized php-1.26wmf24/extensions/Echo/Hooks.php: Remove duplicate 'MediaWiki' prefix from echo.unseen stats (duration: 00m 12s)
* 00:03 RoanKattouw: Running FlowFixLinks.php on testwiki
 
== 2015-09-22 ==
* 23:40 mutante: renaming search mailing lists to discovery mailing lists
* 23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf24/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/240283/ and https://gerrit.wikimedia.org/r/#/c/240281/ (duration: 00m 13s)
* 23:18 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/240278/ (duration: 00m 12s)
* 23:16 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/240259/ (duration: 00m 12s)
* 23:15 logmsgbot: krenair@tin Synchronized w/static/images/sul/wikimania.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 11s)
* 23:14 logmsgbot: krenair@tin Synchronized w/static/images/sul/commons.png: https://gerrit.wikimedia.org/r/#/c/239308/ (duration: 00m 12s)
* 22:44 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgFlowMigrateReferenceWiki to false in production (duration: 00m 12s)
* 22:38 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow opt-in on testwiki for testing (duration: 00m 12s)
* 21:56 cwdent: updated payments from 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3 to 8428499feb8760d63faf681d53995697a2ba0fa7
* 21:49 chasemp: unban elastic1030 from T112559
* 21:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf24
* 21:13 logmsgbot: twentyafterfour@tin Finished scap: Test 1.26wmf24 (duration: 50m 34s)
* 20:44 mutante: cancel backup job of bast1001 on helium because running low on disk
* 20:22 logmsgbot: twentyafterfour@tin Started scap: Test 1.26wmf24
* 16:50 robh: all mw servers returned to puppet enabled, puppet swat window over
* 16:40 awight: updated paymentswiki 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
* 16:23 robh: re-enabled puppet on mw hosts, as both redirection changes are good
* 16:14 robh: re-enabling puppet on mw hosts, as the new patchset 239278 deployed and tested fine on a single host, deploying to rest
* 16:04 robh: disabling puppet across mw hosts for new configuration deployment
* 15:46 godog: running puppet on restbase2001
* 15:41 godog: stop puppet on restbase2* pending codfw expansion
* 15:31 cwdent: updated payments from 153418195a45cab820bc2aacf9a4f7dbc9dde768 to 7b08867d9c5e87f5babb4b5b9cf1f5bec5e243b3
* 14:42 cmjohnson1: shutting down elastic1005 and elastic1030 to move around within the data center
* 14:19 bblack: starting slow restart of varnish + varnish-frontend daemon processes on global text, upload, and mobile clusters for shm_reclen (all randomly blended, no parallelism, ~5 minute spacing, will take ~9 hours - FEs will lose cache data, BEs will not)
* 14:14 chasemp: depool elastic nodes for T112559
* 11:18 logmsgbot: aude@tin Synchronized php-1.26wmf23/extensions/Wikidata: Fix autocomment and change handling bugs (duration: 00m 21s)
* 10:42 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: Enable arbitrary access for Wikibooks (duration: 00m 12s)
* 10:42 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks - try again for snapshot hosts (duration: 00m 12s)
* 10:35 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable data access for Wikibooks (duration: 01m 12s)
* 10:17 moritzm: enabled ferm on mw1152 (videoscaler)
* 10:03 godog: finished stressdisk on restbase200[123] no errors reported
* 10:03 moritzm: enabled ferm on mw1259 (videoscaler)
* 04:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 22 04:35:46 UTC 2015 (duration 35m 45s)
* 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-22 02:22:56+00:00
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 00s)
* 01:53 mutante: sodium - deleted salt key, revoked puppet cert, rm from icinga ..
* 00:32 ori: Disabled Puppet for 24h on hafnium and stopped ganglia-monitor. gmond was saturating CPU.
 
== 2015-09-21 ==
* 23:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/extensions/NavigationTiming/modules/ext.navigationTiming.js: T112593 (duration: 00m 14s)
* 23:21 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/239828/ (duration: 00m 21s)
* 23:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238357/ (duration: 00m 13s)
* 22:26 mutante: restarting gerrit for ssh config change
* 21:15 awight: legacy PayPal listener updated from 1c9ac2e66d11bbf768ea873d6e1a2522ca9841c1 to 55aeef63f6508381e3a8b7fcabddf9a3c3b73b8e
* 20:43 cwdent: updated worldpay config on payments
* 20:41 mdholloway: MobileApps deployed sha1 013044e
* 20:39 chasemp: banning elastic1005 for T112559
* 20:29 subbu: deployed parsoid version 9984d221
* 19:41 urandom: temporarily stopping codfw restbase cassandra nodes to test quorum auth
* 19:15 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Ieccb23f: Enable async secondary writes for mysql-multiwrite cache (on testwiki) (duration: 00m 13s)
* 18:36 ejegg: re-enabled paypal audit parser
* 18:16 cmjohnson1: disabling puppet on mw1031
* 17:58 chasemp: banning 1030 from eqiad elastic cluster for T112559#1660068
* 17:57 ejegg: disabled paypal audit parser
* 16:08 ejegg: updated payments-wiki from 4d9d165c40070e036176dba8987243f6dbc7415e to 153418195a45cab820bc2aacf9a4f7dbc9dde768
* 15:22 logmsgbot: thcipriani@tin Synchronized php-1.26wmf23/extensions/ContentTranslation/modules/entrypoint/ext.cx.interlanguagelink.js: SWAT: Revert "Do not call cxserver to display gray interwiki link" [[gerrit:239819]] (duration: 00m 11s)
* 15:10 logmsgbot: thcipriani@tin Synchronized robots.txt: SWAT: Remove redundant entries from robots.txt [[gerrit:239403]] (duration: 00m 12s)
* 14:33 ottomata: restart eventlogging with mysql consumer replace=True (AKA INSERT IGNORE)
* 14:09 godog: rolling restart restbase in production after cassandra credentials change
* 12:53 godog: rolling restart cassandra after enabling dc encryption, no nodes in codfw yet
* 12:01 moritzm: repooled mw1160 (for T104969)
* 11:54 moritzm: depooled mw1160 (for T104969)
* 11:51 moritzm: repooled mw1158, mw1159 (for T104969)
* 11:39 moritzm: depooled mw1158, mw1159 (for T104969)
* 11:37 moritzm: depooled and repooled mw1156, mw1157 (for T104969)
* 11:26 moritzm: repooled mw1154, mw1155 (for T104969)
* 11:21 moritzm: depooled mw1154, mw1155 (for T104969)
* 10:39 moritzm: repooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
* 10:24 moritzm: depooled mw1026-mw1029 and mw1110-mw1113 (for T104968)
* 10:17 moritzm: repooled mw1100-mw1109 (for T104968)
* 10:17 godog: create restbase user on cassandra cluster
* 10:06 moritzm: depooled mw1100-mw1109 (for T104968)
* 09:56 moritzm: repooled mw1140 and mw1142-mw1148 (for T104968)
* 09:41 moritzm: depooled mw1140 and mw1142-mw1148 (for T104968)
* 09:36 moritzm: repooled mw1130-mw1139 (for T104968)
* 09:22 moritzm: depooled mw1130-mw1139 (for T104968)
* 09:14 moritzm: repooled mw1120-mw1129 (for T104968)
* 09:02 moritzm: depooled mw1120-mw1129 (for T104968)
* 08:48 moritzm: repooled mw1189 and mw1200-mw1208 (for T104968)
* 08:33 moritzm: depooled mw1189 and mw1200-mw1208 (for T104968)
* 08:29 godog: switch to 'restbase' cassandra user on restbase test cluster
* 08:29 moritzm: repooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
* 08:21 _joe_: restarted the logstash agent on logstash1003, OOM'd
* 08:18 moritzm: depooled mw1190-mw1195 and mw1197-mw1199 (for T104968)
* 08:07 _joe_: installing the new HHVM package on the api canaries
* 08:04 moritzm: repooled mw1221-mw1229 (for T104968)
* 07:53 moritzm: depooled mw1221-mw1229 (for T104968)
* 07:49 moritzm: repooled mw1230-mw1235 (for T104968)
* 07:43 _joe_: installing the new hhvm package on the canary appservers
* 07:08 moritzm: depooled mw1230-mw1235 (for T104968)
* 04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 21 04:31:03 UTC 2015 (duration 31m 2s)
* 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-21 02:23:12+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 25s)
* 02:06 MaxSem: Maps: created indexes on admin. <3 Postgres :(
* 01:56 bblack: downtimed eqiad ipv6 text/upload alerts as well, as with mobile above ( 1 301 TLS Redirect - 505 bytes in 1.008 second response time
* 01:46 bblack: downtimed the "LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6" alert for now ( https://phabricator.wikimedia.org/T113154 )
 
== 2015-09-20 ==
* 22:34 yuvipanda: reloda pybal on lvs1012
* 17:01 bblack: repooling cp1046 varnish-be + varnish-be-rand in confctl, fresh storage, purge queue caught up - T113184
* 16:44 bblack: depooling cp1046 varnish-be + varnish-be-rand in confctl, wiping storage, re-pooling - T113184
* 07:24 paravoid: temporarily disabling puppet on fermium and applying antispam countermeasures
* 04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 20 04:29:16 UTC 2015 (duration 29m 15s)
* 02:22 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-20 02:22:53+00:00
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 12s)
 
== 2015-09-19 ==
* 23:12 urandom: begining Cassandra repair on restbase1005 (nodetool repair -pr)
* 23:08 urandom: begining Cassandra repair on restbase1004 (nodetool repair -pr)
* 19:56 jynus: restarting once more giblit, last chance
* 19:04 paravoid: salt rm /etc/systemd/system/txstatsd.service from all cp*, leftover because of ::txstatsd::decommission (removed with 4a1d4e) missing it
* 19:00 ejegg|away: updated SmashPig from d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85 to d1baa32267eaad7d69b47c657f4853eb306fad6b
* 18:45 _joe_: restarted gitblit. I will now substitute myself with a clever perl one-liner.
* 18:38 paravoid: pooling back cp1046 to pybal eqiad/mobile, has stayed stable
* 18:34 paravoid: reactivating ΒGP with GTT @ eqiad
* 08:42 _joe_: cp1046 dead on console again, powercycling to inspect it
* 05:49 logmsgbot: aaron@tin Synchronized php-1.26wmf23/extensions/TitleBlacklist: 80d3a21a51f9c54ed2d94 (duration: 00m 12s)
* 05:22 paravoid: pybal-depooling cp1046 from eqiad/mobile until further investigation
* 05:21 paravoid: powercycling cp1046, dead on console
* 05:01 awight: deploy SmashPig config to limit weekend spam
* 04:40 awight: update crm from 15ea14f61338ca9f34e9ccb9f56eae14a161380a to 9fa38d06a75363a8009bce7ced190e39c75b68bc
* 04:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 19 04:28:59 UTC 2015 (duration 28m 57s)
* 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-19 02:23:29+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 05s)
 
== 2015-09-18 ==
* 23:22 awight: update fundraising-tools from 3e0e3ae799a507b378d0ece3e71631b10b361329 to e1b60fa2c258fd4ff55905b03a4d8886132278c1
* 20:52 ebernhardson: restart es on elastic1025 to disable dynamic scripting
* 20:34 gwicke: dropped by_ns indexes on restbase title_revisions tables
* 19:54 gwicke: finished deploy of restbase daacf4daa
* 19:45 gwicke: re-enabled puppet on restbase100*
* 19:35 gwicke: canary deploy of restbase daacf4daa on restbase1001; moving forward so that we can re-enable puppet over the weekend.
* 18:38 cwdent: updated payments from 1bdd287b083032ff418434ad6bb6920735af918a to 4d9d165c40070e036176dba8987243f6dbc7415e
* 17:54 logmsgbot: ebernhardson@tin Synchronized wmf-config/CommonSettings.php: Replace insecure es usage with usage of a plugin (duration: 00m 12s)
* 16:41 mutante: mailman now on 2.1.18 and jessie
* 16:14 dcausse: elastic in eqiad plugin updates: restarting elastic1021
* 16:07 paravoid: deactivating ΒGP with GTT @ eqiad
* 15:20 godog: create restbase user on cassandra test cluster
* 14:55 dcausse: elastic in eqiad plugin updates: restarting elastic1020
* 14:22 bblack: committing lvs1007-1012 port/vlan changes for asw-d-eqiad (but leaving all 6 LVS ports in "disabled" state - T112781 )
* 14:14 bblack: committing lvs1007-12 port/vlan changes for asw-b-eqiad, round 3...
* 14:11 mutante: sodium - stopped exim - rsyncing lists to fermium
* 14:10 dcausse: elastic in eqiad plugin updates: restarting elastic1019
* 14:07 mutante: stopped mailman on sodium
* 14:01 bblack: rollback on asw-b-eqiad changes above
* 13:56 bblack: committing eqiad lvs1007-1012 port/vlan changes for asw-b-eqiad
* 13:20 bblack: committing eqiad lvs1007-12 port/vlan changes for asw-c-eqiad
* 13:16 bblack: commiting eqiad lvs1007-12 port/vlan changes for asw2-a5-eqiad
* 13:12 dcausse: elastic in eqiad plugin updates: restarting elastic1018
* 12:21 godog: restart logstash on logstash1001, OOM in logs
* 11:55 dcausse: elastic in eqiad plugin updates: restarting elastic1017
* 11:06 dcausse: elastic in eqiad plugin updates: restarting elastic1016
* 10:28 moritzm: restarted salt-master on palladium
* 09:46 moritzm: installed openldap security updates on plutonium
* 09:37 moritzm: installed openldap security updates on pollux
* 09:33 dcausse: elastic in eqiad plugin updates: restarting elastic1015
* 08:22 dcausse: elastic in eqiad plugin updates: restarting elastic1014
* 07:21 dcausse: elastic in eqiad plugin updates: restarting elastic1013
* 06:15 dcausse: elastic in eqiad plugin updates: restarting elastic1012
* 04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 18 04:37:42 UTC 2015 (duration 37m 41s)
* 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-18 02:31:49+00:00
* 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 08s)
* 02:21 logmsgbot: krenair@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 12s)
* 02:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/218353/ (duration: 00m 11s)
* 02:13 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237149/ (duration: 00m 12s)
* 02:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234544/ (duration: 00m 12s)
* 01:58 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 12s)
* 01:58 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: ResourceLoaderModule: cache file content hash (duration: 00m 11s)
* 01:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://phabricator.wikimedia.org/T106264 (duration: 00m 12s)
* 01:36 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237331/ (duration: 00m 12s)
* 00:14 ori: restarted logstash on logstash1001
 
== 2015-09-17 ==
* 23:39 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/238978/ (duration: 00m 12s)
* 23:05 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
* 23:04 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 12s)
* 22:53 gwicke: puppet on restbase cluster disabled since about  21:30 UTC for gradual deploy; ran into minor issue in staging, which is now being addressed, after which deploy will continue
* 21:22 logmsgbot: ori@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 13s)
* 21:22 logmsgbot: ori@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoaderModule.php: I952068d2d: Use MD4 to compute file hash rather than SHA1 (duration: 00m 12s)
* 20:44 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
* 20:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/239206/ (duration: 00m 12s)
* 19:46 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
* 19:41 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/239181/ (duration: 00m 14s)
* 19:12 mutante: powercycling unresponse mw1005
* 18:14 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf23
* 17:38 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 13s)
* 17:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/includes/registration/ExtensionRegistry.php: registration: Fix merging of array_plus (duration: 00m 11s)
* 16:43 chasemp: restart elasticsearch on 1005
* 16:16 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235900/ (duration: 00m 12s)
* 15:15 dcausse: elastic in eqiad plugin updates: restarting elastic1004 (take 2)
* 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable Suggestions in ptwiki [[gerrit:238097]] (duration: 00m 13s)
* 14:22 mutante: analytics1029 -  Failed to start Hadoop datanode
* 14:20 mutante: starting hadoop datanode on analytics1029
* 14:14 _joe_: reimaging tmh1001 to mw1259
* 14:11 jynus: stopping replication and applying schema change to db1051
* 14:05 dcausse: elastic in eqiad plugin updates: can't restart elastic1004 (2 timeouts when disabling replication, too much load?), waiting for more shards to rebalance...
* 13:58 dcausse: elastic in eqiad plugin updates: restarting elastic1004
* 13:50 moritzm: repooled mw1236-mw1239 (T104968)
* 13:34 moritzm: depooled mw1236-mw1239 (T104968)
* 13:26 moritzm: repooled mw1090-mw1099 (T104968)
* 13:16 moritzm: depooled mw1090-mw1099 (T104968)
* 13:13 moritzm: repooled mw1080-mw1089 (T104968)
* 13:05 moritzm: depooled mw1080-mw1089 (T104968)
* 13:01 moritzm: repooled mw1070-mw1079 (T104968)
* 12:49 moritzm: depooled mw1070-mw1079 (T104968)
* 12:35 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
* 12:24 moritzm: depooled mw1060 and mw1062-mw1069 (T104968) (not repooled)
* 12:24 moritzm: repooled mw1060 and mw1062-mw1069 (T104968)
* 12:16 moritzm: repooled mw1050-mw1059
* 12:04 moritzm: depooled mw1050-mw1059
* 11:39 moritzm: repooled mw1040 and mw1042-mw1049 (T104968)
* 11:36 dcausse: elastic in eqiad plugin updates: restarting elastic1003
* 11:26 moritzm: typoed earlier entry: "mw1032-mw1039" instead of "mw1032-mw1239"
* 11:26 moritzm: depooled mw1040 and mw1042-mw1049 (T104968)
* 11:18 moritzm: repooled mw1030 and mw1032-mw1239 (T104968)
* 11:03 moritzm: depooled mw1030 and mw1032-mw1239 (T104968)
* 10:35 moritzm: repooled mw1250-mw1258 (T104968)
* 10:27 moritzm: depooled mw1250-mw1258 (T104968)
* 10:25 _joe_: killing temporarily subra
* 10:24 moritzm: repooled mw1240-mw1249 (T104968)
* 10:19 _joe_: experimenting with poolcounter issues on subra
* 10:18 logmsgbot: oblivian@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: Use codfw poolcounters in codfw (duration: 00m 12s)
* 10:12 moritzm: depooled mw1240-mw1249 (T104968)
* 10:12 dcausse: elastic in eqiad plugin updates: restarting elastic1002
* 10:05 logmsgbot: hoo@tin Synchronized wmf-config/: Set 'repoConceptBaseUri' for all Wikibase clients (duration: 00m 13s)
* 10:00 dcausse: elastic in eqiad plugin updates: unfreezing indices
* 09:48 dcausse: elastic in eqiad plugin updates: no more groovy in warmers, waiting for few more shards to move in elastic1001 and will unfreeze indices to test warmers
* 09:39 dcausse: elastic in eqiad plugin updates: deleting warmers manually for old unused indices (eswikisource_content_1415240352, ruwiki_content_1415302164, thwiki_content_1415318677). We will have to remove these indices.
* 09:39 paravoid: repooling ulsfo US-West traffic back to ulsfo for the first time since May :)
* 09:01 dcausse: elastic in eqiad plugin updates: updating warmers on all wikis
* 08:58 paravoid: penalizing ulsfo-eqiad direct MPLS links to higher OSPF weights
* 08:57 paravoid: adjusting OSPF weights to be latency-based across the US network
* 08:53 _joe_: removed iptables rules for dropping traffic to helium on mw1017
* 08:52 dcausse: elastic in eqiad plugin updates: index warmer queries are outdated with inline groovy script, updating warmers on warwiki first to test
* 08:05 paravoid: eqiad-codfw -> eqiad-eqord-codfw migration
* 07:49 moritzm: repooled mw1180-mw1188 (T104968)
* 07:42 dcausse: elastic in eqiad plugin updates: restarting elastic1001
* 07:42 moritzm: depooled mw1180-mw1188 (T104968)
* 07:37 moritzm: repooled mw1170-mw1179 (T104968)
* 07:36 dcausse: elastic in eqiad plugin updates: freezing indices
* 07:27 moritzm: depooled mw1170-mw1179 (T104968)
* 07:14 _joe_: uploading new HHVM package
* 07:07 moritzm: repooled mw1161-1168 (T104968)
* 06:57 moritzm: depooled mw1161-1168 (T104968)
* 06:45 moritzm: repooled mw1209-mw1220 with ferm enabled
* 06:33 moritzm: depooling mw1209-mw1220 (in two steps)
* 05:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 17 05:47:47 UTC 2015 (duration 47m 46s)
* 03:06 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-17 03:06:33+00:00
* 03:03 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s)
* 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-17 02:45:48+00:00
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 11s)
* 00:35 cwdent: updated payments from 155cdeb737c01baf62551292764fd2f5a93a9a63 to 1bdd287b083032ff418434ad6bb6920735af918a
 
== 2015-09-16 ==
* 23:27 bblack: updating eqiad switch configs for lvs1007-1012 vlan/trunk settings
* 23:19 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/MobileFrontend/resources/mobile.overlays/Overlay.less: https://gerrit.wikimedia.org/r/#/c/238865/ (duration: 00m 11s)
* 23:13 gwicke: started `nodetool rebuild -- eqiad` on restbase-test200{1,2
* 23:03 cwdent: updated payments from 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1 to 155cdeb737c01baf62551292764fd2f5a93a9a63
* 22:26 ejegg: updated SmashPig from fdb053efa617162ac9f695e493c390987a069140 to d5895428d1d8ebc5a6e172e8cdec6dbec0b10d85
* 22:08 urandom: disabling puppet in RESTBase eqiad staging cluster to test new code and config
* 22:08 ottomata: powercycling  analytcis1029, it is down?
* 20:47 cscott: updated OCG to version 4032a596ce6eb442b02cc6ee9b79263b1eb23275
* 19:42 ejegg: updated crm from abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b to 15ea14f61338ca9f34e9ccb9f56eae14a161380a
* 19:38 ori: Deployed statsv 0bfd9f06f / change I050a12d3b
* 18:47 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf23
* 18:38 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf23: syncing wmf23 ahead of deployment to group1 (duration: 01m 35s)
* 17:34 paravoid: asw-d-eqiad: toggling RE mastership again
* 17:26 godog: stop puppet on restbase* to apply https://gerrit.wikimedia.org/r/#/c/238738/ / merge / reenable puppet
* 16:54 _joe_: turned on the hhvm tmh, stopping the zend ones for testing
* 16:44 logmsgbot: oblivian@tin Synchronized wmf-config/CommonSettings.php: use ffmpeg whereever possible (duration: 00m 12s)
* 16:16 bblack: upgrading pybal on lvs400[12]
* 16:12 bblack: upgrading pybal on lvs400[34], lvs300[34]
* 16:08 bblack: upgrading pybal on lvs200[123]
* 16:05 bblack: upgrading pybal on lvs200[456]
* 15:44 _joe_: uploading pybal 1.10 to reprepro, installing to the test cluster
* 15:24 moritzm: uploaded debdeploy 0.0.6 to apt.wikimedia.org
* 15:10 hashar: Started using Nodepool spawned instances.  Moved integration-jjb-config-diff Jenkins job to Nodepool with https://gerrit.wikimedia.org/r/#/c/238752/  . See also: https://phabricator.wikimedia.org/T112750
* 15:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Add m.wikidata.org to wgCrossSiteAJAXdomains (duration: 00m 12s)
* 14:51 _joe_: experimenting on testwiki for poolcounter failure scenarios
* 14:45 moritzm: enabled ferm on mw1010 (jobrunner) in eqiad
* 14:27 paravoid: asw-d-eqiad: toggling RE mastership
* 14:18 paravoid: disabling/ignoring asw-d-eqiad @ librenms
* 14:09 jynus: upgrading and restarting db1051
* 13:57 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1051 for maintenance (duration: 00m 12s)
* 13:40 urandom: initiating Cassandra repair on restbase1007 (nodetool repair -pr)
* 13:40 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 01m 37s)
* 13:35 moritzm: repooled mw1149-mw1151 (with ferm enabled)
* 13:24 moritzm: depooled mw1149-mw1151 (for enabling ferm)
* 13:19 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Reverting depool of es1055 (duration: 00m 12s)
* 13:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1055 for maintenance (duration: 00m 12s)
* 13:03 paravoid: disabling asw-d-eqiad xe-8/0/23, xe-8/0/24, xe-8/0/25, xe-8/0/26, xe-8/0/27, xe-8/0/28; servers reboot-looping -> asw-d's SNMP unhappy -> librenms unhappy -> faidon's mailbox unhappy
* 12:48 moritzm: repooled mw1115-mw1117, mw1119 (with ferm enabled)
* 12:42 moritzm: depooling mw1115-mw1117, mw1119 (mw1118 was already depooled) to enable ferm
* 11:32 moritzm: repooled mw1019-mw1025 with ferm enabled
* 11:24 jynus: making db1069 a sibling of db1055 (s1)
* 11:13 godog: create restbase user on cassandra test cluster
* 11:07 moritzm: depooled mw1019-mw1025 (to enable ferm)
* 10:52 logmsgbot: catrope@tin Synchronized php-1.26wmf23: (no message) (duration: 02m 04s)
* 10:49 logmsgbot: catrope@tin Synchronized php-1.26wmf22: (no message) (duration: 02m 12s)
* 10:48 jynus: reenabling semisync on db1072 and db1073
* 10:47 logmsgbot: catrope@tin scap aborted: (no message) (duration: 00m 21s)
* 10:47 logmsgbot: catrope@tin Started scap: (no message)
* 10:24 logmsgbot: catrope@tin Synchronized php-1.26wmf23/includes/changes/EnhancedChangesList.php: T112738 (duration: 00m 12s)
* 10:09 logmsgbot: aude@tin Synchronized arbitraryaccess.dblist: (no message) (duration: 00m 11s)
* 09:37 awight: ruthlessly disabled PayPal IPN listener
* 08:12 moritzm: repooled mw1153 with ferm enabled
* 07:57 jynus: truncated some tables from ContentTranslation extension on x1
* 07:57 moritzm: depooled mw1153 (it's an image scaler, of course) to enable ferm
* 07:56 moritzm: depooled mw1153 (videoscaler) to enable ferm
* 06:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 16 06:31:58 UTC 2015 (duration 31m 57s)
* 03:28 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Don't waste milliseconds counting microseconds (duration: 00m 12s)
* 03:27 logmsgbot: ori@tin Synchronized php-1.26wmf23/vendor/monolog/monolog/src/Monolog/Logger.php: Iccfda47689: monolog: Dont waste milliseconds counting microseconds ; sync-file php-1.26wmf22/vendor/monolog/monolog/src/Monolog/Logger.php Iccfda47689: monolog: Dont waste milliseconds counting microseconds (duration: 00m 12s)
* 03:12 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf23) at 2015-09-16 03:12:08+00:00
* 03:05 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 10m 30s)
* 02:38 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-16 02:38:48+00:00
* 02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 07m 02s)
* 01:03 logmsgbot: krinkle@tin Synchronized php-1.26wmf23/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 12s)
* 00:29 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: hotfix Ia2fcd13f4 (duration: 00m 11s)
* 00:15 logmsgbot: legoktm@tin Synchronized php-1.26wmf23/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
* 00:14 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/CentralAuth/includes/: Use set() for tokens with unique keys (duration: 00m 12s)
* 00:11 bblack: reinstalling lvs400[12] to jessie (traffic on 400[34], already jessie)
* 00:08 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/VisualEditor/modules/ve-mw/ui/styles/dialogs: https://gerrit.wikimedia.org/r/#/c/238646/ (duration: 00m 12s)
 
== 2015-09-15 ==
* 23:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238617/ (duration: 00m 12s)
* 23:48 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: https://gerrit.wikimedia.org/r/#/c/238618/ (duration: 00m 12s)
* 23:42 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 14s)
* 23:42 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238543/ (duration: 00m 12s)
* 23:40 logmsgbot: krenair@tin Synchronized php-1.26wmf22/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238544/ (duration: 00m 11s)
* 23:38 logmsgbot: krenair@tin Synchronized php-1.26wmf23/includes/resourceloader/ResourceLoader.php: https://gerrit.wikimedia.org/r/#/c/238545/ (duration: 00m 11s)
* 23:24 yurik: deployed kartotherian & tilerator
* 23:22 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238512/ (duration: 00m 12s)
* 23:16 logmsgbot: krenair@tin Synchronized php-1.26wmf23/extensions/EventLogging/modules/ext.eventLogging.core.js: https://gerrit.wikimedia.org/r/#/c/238513/ (duration: 00m 12s)
* 21:15 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: touch files edited in I0cb6fe37e and re-sync to cluster (duration: 00m 13s)
* 21:13 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf23
* 21:10 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded (duration: 03m 52s)
* 21:07 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, once more because mw1010 overloaded
* 21:05 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf23 to testwiki, again (duration: 47m 49s)
* 20:47 mutante: mw1010 - extremely slow,finally got on and attempted to restart hhvm. load going down
* 20:17 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki, again
* 20:17 logmsgbot: twentyafterfour@tin scap aborted: sync 1.26wmf23 to testwiki (duration: 82m 58s)
* 20:05 ottomata: restarted mysql (and oozie) on analytics1027 to start mysql binlogging
* 18:54 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf23 to testwiki
* 16:55 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1003, es1004, es1007 and es1010 for decommision (duration: 00m 12s)
* 16:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Revert depool db1055 for maintenance (duration: 00m 11s)
* 16:39 ottomata: reinstalling analytics1015
* 16:32 RoanKattouw: Putting wmf22 versions of Echo and MobileFrontend on mw1017 for testing
* 16:30 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/WikimediaEvents.php: touch file that is serving old version in prod (duration: 00m 12s)
* 16:29 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSuggest.js: Touch file that is serving old version in prod (duration: 00m 12s)
* 16:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 for maintenance (duration: 00m 11s)
* 16:11 bblack: traffic DNS depooled out of codfw for now T112639
* 15:38 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: CX: Enable suggestion for testwiki (part 2) [[gerrit:237327]] (duration: 00m 13s)
* 15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable suggestion for testwiki (part 1) [[gerrit:237327]] (duration: 00m 12s)
* 15:31 logmsgbot: thcipriani@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/jquery/jquery.mwCoolCats.js: SWAT: Do not fail horribly when invalid categories are passed [[gerrit:238421]] (duration: 00m 12s)
* 15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-eqiad.php: SWAT: poolcounter: enable connect_timeout for testwiki [[gerrit:238109]] (duration: 00m 19s)
* 15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/PoolCounterSettings-codfw.php: SWAT: poolcounter: add connect_timeout in codfw [[gerrit:238108]] (duration: 00m 12s)
* 15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase.php: SWAT: Exclude Flow topic boards and Draft NS from Special:UnconnectedPages [[gerrit:229197]] (duration: 00m 11s)
* 14:51 godog: bounce cassandra on test cluster to deploy  https://gerrit.wikimedia.org/r/236391
* 14:22 cmjohnson1: swapped disk on db1043
* 13:12 moritzm: repool mw1114 (with ferm enabled)
* 13:11 bblack: failing over LVS service in ulsfo to secondariess (400[12] pybal stopped, traffic on jessie-based 400[34])
* 12:53 moritzm: depooled mw1114 (for enabling ferm)
* 11:42 moritzm: repool mw1018 (with ferm enabled)
* 11:23 moritzm: depooled mw1018 (for enabling ferm)
* 08:53 _joe_: created a 100 G partition on a LV on copper, for /tmp
* 08:24 godog: bounce ms-be2006, xfs
* 08:22 moritzm: bumped default size of iptables connection tracking table to 256k
* 06:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 15 06:10:52 UTC 2015 (duration 10m 51s)
* 02:46 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-15 02:46:50+00:00
* 02:40 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 53s)
* 02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/MobileFrontend: Revert Echo to 1.26wmf21 state (duration: 00m 11s)
* 02:18 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo: Revert Echo to 1.26wmf21 state (duration: 00m 12s)
* 01:30 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src: T112287 (duration: 00m 11s)
* 00:49 bblack: reinstalling lvs300[34] to jessie
* 00:43 logmsgbot: ebernhardson@tin Synchronized wmf-config/CirrusSearch-labs.php: noop sync of labs config change (duration: 00m 11s)
* 00:03 logmsgbot: tstarling@tin Synchronized php-1.26wmf22/extensions/ParsoidBatchAPI: for I56d28e9a for RT testing, not live yet (duration: 00m 13s)
 
== 2015-09-14 ==
* 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: Change bucket selection methods in CompletionSuggestions AB test (duration: 00m 12s)
* 23:23 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf22/extensions/UploadWizard/: Swat out badtoken fix to UploadWizard in 1.26wmf22 (duration: 00m 12s)
* 22:37 yurik: deployed tilerator
* 21:15 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: Ie44fcb500: Avoid checking blacklists in isBlacklisted() for existing titles (duration: 00m 12s)
* 21:15 mutante: labnodepool1001 - re-enable puppet and nodepool
* 20:59 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Hack around OOUI's icon pack being too large by creating our own (duration: 00m 12s)
* 20:53 cscott: updated OCG to version 5811056e28f2bc6408b6da96095352ab381bb11f
* 20:21 andrewbogott: graceful’d apache2 on labcontrol1001
* 20:15 subbu: deployed parsoid sha 3d5f4359
* 19:25 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Only load nojs Special:Notifications styles on the special page (duration: 00m 12s)
* 18:05 urandom: rebuilding restbase-test2001.codfw (nodetool rebuild -- eqiad)
* 16:12 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: For real this time (duration: 00m 11s)
* 16:06 ottomata: stopping hdfs journalnode on analytics1011 to copy journal edits to new journalnodes on analytics1035 and analytics1052
* 15:46 godog: switch to openjdk-8 and bounce cassandra on restbase-test200*
* 15:39 bblack: reinstalling lvs4003, lvs4004 (jessie upgrade: T96375) (typo earlier)
* 15:39 bblack: reinstalling lvs4003, lvs4003 (jessie upgrade: T96375)
* 15:34 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 13s)
* 15:05 logmsgbot: krenair@tin Synchronized .gitignore: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 13s)
* 15:05 logmsgbot: krenair@tin Synchronized docroot/noc: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 12s)
* 15:04 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/237529/ (duration: 00m 11s)
* 15:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234980/ (duration: 00m 12s)
* 13:38 godog: stop puppet on restbase-test2001 and turn up cassandra
* 12:56 bblack: rebooting lvs2006 to test eth hw params stuff...
* 12:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/238125/ (duration: 00m 13s)
* 12:50 urandom: starting Cassandra repair on restbase1003 (nodetool repair -pr)
* 12:32 godog: enable dc encryption on cassandra test cluster and rolling restart
* 11:33 mobrovac: citoid deploying d569951
* 10:35 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002, es1005, es1008 (duration: 00m 12s)
* 10:04 jynus: db1029 (x1-master) temporarily saturated by connections- flow was unresponsive for 10 minutes; migration partially aborted
* 09:08 jynus: applying schema change to flowdb
* 08:52 godog: rename cassandra test cluster and restart
* 08:44 godog: silence mendelevium for today, status unclear T111532
* 08:30 jynus: endinf profiling and executing pt-query-digest on db1043 [ETA:4h]
* 07:52 godog: reboot ms-be1010 to pick up disk ordering change
* 04:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 14 04:47:58 UTC 2015 (duration 47m 57s)
* 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-14 02:29:48+00:00
* 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 59s)
* 01:31 Krinkle: mwscript deleteEqualMessages.php --wiki sqwiki
 
== 2015-09-13 ==
* 06:02 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 13 06:02:52 UTC 2015 (duration 2m 51s)
* 02:40 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-13 02:40:43+00:00
* 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 10m 13s)
 
== 2015-09-12 ==
* 20:15 ori: Rolling back Echo to 1.26wmf21 branch on mw1017 (testwiki) to measure increase in render-blocking CSS size
* 19:21 urandom: performing Cassandra repair on restbase1002 (nodetool repair -pr)
* 14:50 jynus: phab.wmfusercontent.org has been temporarily switched to phab.wikivoyage.org due to cert issues
* 04:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 12 04:52:01 UTC 2015 (duration 52m 0s)
* 02:35 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-12 02:35:36+00:00
* 02:32 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 54s)
 
== 2015-09-11 ==
* 21:21 hashar: shutdown nodepool on labnodepool1001.eqiad.wmnet until monday
* 18:01 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression fixes #2 (duration: 00m 12s)
* 16:43 logmsgbot: krinkle@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: T112232 (duration: 00m 12s)
* 16:37 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/: Echo regression backports (duration: 00m 12s)
* 16:35 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (again) (duration: 00m 13s)
* 16:33 legoktm: ssh: connect to host mw1156.eqiad.wmnet port 22: Connection timed out
* 16:32 paravoid: powercycling mw1156, multiple kernel backtraces in console output
* 16:32 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/resources/src/mediawiki/mediawiki.js: resourceloader: Document internal mw.loader#jobs property (duration: 01m 07s)
* 16:15 cmjohnson1: mw1031 rebooting for f/w update
* 16:07 bblack: enabled LRO+GRO on lvs200[123], starting pybal there again ([456] testing looks good so far)
* 15:45 bblack: enabled LRO+GRO on lvs200[456] (backups).  Stopping pybal on lvs200[123] to test...
* 15:11 cmjohnson1: swapping pem2 cr2-eqiad
* 10:03 jynus: starting nodepool in labnodepool1001
* 09:21 jynus: starting profiling of phabricator db (db1043). Very low overhead.
* 06:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 11 06:03:00 UTC 2015 (duration 2m 59s)
* 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-11 02:41:24+00:00
* 02:34 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 11m 18s)
* 01:16 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/TitleBlacklist: 9bf13dbe0b, 3203b045f7 (duration: 00m 12s)
 
== 2015-09-10 ==
* 23:52 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237064/ (duration: 00m 11s)
* 23:47 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237056/ (duration: 00m 11s)
* 23:13 logmsgbot: krenair@tin Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/221825 (duration: 00m 13s)
* 23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/224771 (duration: 00m 12s)
* 21:13 logmsgbot: legoktm@tin Synchronized php-1.26wmf22/extensions/Echo/modules: Align popup footer buttons to take 50% width each (duration: 00m 15s)
* 20:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1001; increase weight of es1015 and es1019 (duration: 00m 19s)
* 20:47 ottomata: restarting eventlogging with 12 client side processors on eventlog1001
* 20:31 ottomata: turning off varnishncsa eventlogging eventlistener instances on frontend caches, it is now superseded by varnishkafka
* 20:28 mutante: killed/restarted ganglia aggregator process for mobile-cache, upload cache, misc esams ...
* 20:22 jynus: last SCAP failed on 266/466 hosts
* 20:21 mutante: killed/restarted ganglia aggregator process for text-caches esams on hooft
* 20:17 yurik: deployed kartotherian
* 20:08 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001; increase weight of es1015 and es1019 (duration: 00m 11s)
* 19:11 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf22
* 19:09 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/CentralNotice: deploy https://gerrit.wikimedia.org/r/#/c/237458/ (duration: 00m 12s)
* 18:57 twentyafterfour: restarted phd on iridium
* 18:51 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22/extensions/Wikidata: Deploy wikidata patch: https://gerrit.wikimedia.org/r/#/c/237449/ (duration: 00m 19s)
* 18:23 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf22: deploy https://gerrit.wikimedia.org/r/#/c/237440/ (duration: 01m 42s)
* 18:09 cmjohnson1: reseating pem2 cr2-eqiad
* 16:52 akosiaris: puppetswat done
* 16:50 mobrovac: restbase rolling restart of rb100x
* 16:49 mobrovac: restbase enabled puppet on rb100x
* 16:13 akosiaris: started puppetSWAT
* 16:10 logmsgbot: marktraceur@tin Finished scap: Make sure codfw got the last few patches sync'd to it (duration: 07m 36s)
* 16:03 logmsgbot: marktraceur@tin Started scap: Make sure codfw got the last few patches sync'd to it
* 16:02 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/: [SWAT] [wmf22] Revert opera redirect loop fix that caused redirect loops in Firefox (duration: 02m 30s)
* 15:55 mobrovac: restbase disabled puppet on rb100x
* 15:45 logmsgbot: marktraceur@tin Synchronized php-1.26wmf22/extensions/UploadWizard/resources/transports/mw.FormDataTransport.js: [SWAT] [wmf22] Always set 'offset' with chunked uploads, even for first chunk (offset == 0) (duration: 02m 21s)
* 15:26 ottomata: started hadoop decomission of analytics1016
* 15:21 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] Attempting another sync to mw2187 hoping it's up now (duration: 02m 22s)
* 15:05 logmsgbot: marktraceur@tin Synchronized wmf-config/: [SWAT] [config] Beta: Enable Content Translation suggestions (duration: 02m 22s)
* 13:35 moritzm: enabled ferm on mediawiki app servers in codfw
* 13:30 jynus: performing schema change and maintenance on officewiki and public all wikis with flow enabled
* 12:51 moritzm: enabled ferm on mediawiki API servers in codfw
* 12:36 moritzm: enabled ferm on mediawiki video scalers, image scalers and job runners in codfw
* 09:20 mobrovac: restbase deploying 0182962
* 06:13 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 10 06:13:14 UTC 2015 (duration 13m 13s)
* 03:02 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-10 03:02:45+00:00
* 02:59 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 06m 10s)
* 02:51 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237304 (duration: 00m 11s)
* 02:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance/dumpInterwiki.php: https://gerrit.wikimedia.org/r/237303 (duration: 00m 10s)
* 02:43 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-10 02:43:20+00:00
* 02:36 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 10m 45s)
* 02:24 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.js: Ic0b1fb64ee7 backport (duration: 00m 12s)
* 01:04 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 13s)
* 01:03 logmsgbot: ori@tin Synchronized php-1.26wmf22/extensions/NavigationTiming: I2605c746b: Ensure timings are reported after the page has loaded (duration: 00m 12s)
* 00:54 mutante: powercycling unresponsive mw1154
 
== 2015-09-09 ==
* 23:34 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
* 23:31 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
* 23:29 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
* 23:23 MaxSem: deployed Kartotherian config updates
* 23:23 logmsgbot: catrope@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 11s)
* 23:22 RoanKattouw: Running updateinterwikicache
* 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaMaintenance: SWAT (duration: 00m 13s)
* 23:13 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Flow: SWAT (duration: 00m 32s)
* 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/WikimediaMaintenance: SWAT (duration: 00m 14s)
* 23:12 logmsgbot: catrope@tin Synchronized php-1.26wmf21/extensions/Flow: SWAT (duration: 00m 29s)
* 20:17 subbu: deployed parsoid version ffd0b444
* 18:15 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf22
* 16:47 andrewbogott: systemctl stop nodepool on labnodepool1001
* 16:06 logmsgbot: aude@tin Synchronized database lists: Remove unused usagetracking.dblist (duration: 00m 12s)
* 16:01 logmsgbot: krenair@tin Synchronized robots.txt: https://gerrit.wikimedia.org/r/#/c/236200/ (duration: 00m 12s)
* 15:57 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236701/ - noop (duration: 00m 12s)
* 15:56 ejegg: updated payments from from 4c5e30288370db926cbbf7a7528edb9c41c65716 to 9fc8ab40b7f70c7b588c2b9e7b5c94b1f893faa1
* 15:50 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237104/ (duration: 00m 12s)
* 15:46 logmsgbot: krenair@tin Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
* 15:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/237097/ (duration: 00m 12s)
* 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/resources/src/mediawiki/mediawiki.searchSuggest.js: Enable completion suggester AB experiment (duration: 00m 12s)
* 15:43 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/: Enable suggester AB experiement (duration: 00m 11s)
* 15:38 logmsgbot: krenair@tin Synchronized php-1.26wmf22/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/237091/ (duration: 00m 21s)
* 15:26 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234425/ (duration: 00m 12s)
* 15:21 logmsgbot: krenair@tin Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/236994/ (duration: 00m 12s)
* 15:15 bd808: Running sync-common manually on mw2187.codfw.wmnet. Host is missing l10n cache files
* 15:12 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236025/ (duration: 00m 11s)
* 15:10 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236042/ (duration: 00m 13s)
* 14:03 mutante: beginning mailman migration - expect lists to be down
* 13:14 moritzm: enabled ferm on test.wikipedia.org (mw1017)
* 13:05 urandom: issuing Cassandra repair on restbase1001 (nodetool repair -pr)
* 13:02 moritzm: enabled ferm on various initial mediawiki hosts in codfw: videoscaler (mw2007), appserver (mw200[89]), jobrunner (mw2081), api (mw2050), imagescaler (mw2086)
* 10:33 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Remove unused usagetracking tag (duration: 00m 11s)
* 10:30 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 12s)
* 10:26 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: rv usage tracking (duration: 00m 12s)
* 10:23 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on commons and test2wiki (duration: 00m 11s)
* 10:21 logmsgbot: aude@tin Synchronized wikidataclient.dblist: Sorted dblist (duration: 00m 12s)
* 09:41 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikinews (duration: 00m 12s)
* 08:35 moritzm: installed spice security updates on labvirt*, ganeti* and labnodepool1001
* 05:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep  9 05:11:28 UTC 2015 (duration 11m 27s)
* 02:55 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf22) at 2015-09-09 02:55:24+00:00
* 02:52 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf22/cache/l10n: l10nupdate for 1.26wmf22 (duration: 05m 34s)
* 02:31 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-09 02:31:50+00:00
* 02:28 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 44s)
* 00:00 logmsgbot: catrope@tin Finished scap: Need to update i18n for a new Echo message (duration: 23m 08s)
 
== 2015-09-08 ==
* 23:36 logmsgbot: catrope@tin Started scap: Need to update i18n for a new Echo message
* 23:36 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: SWAT (duration: 00m 10s)
* 23:36 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT (duration: 00m 13s)
* 23:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: SWAT (duration: 00m 12s)
* 23:33 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: SWAT (duration: 00m 12s)
* 23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/WikimediaEvents/: SWAT (duration: 00m 11s)
* 23:20 logmsgbot: catrope@tin Synchronized php-1.26wmf22/extensions/Echo/: SWAT (duration: 00m 14s)
* 23:14 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 11s)
* 22:13 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: re-apply patch 1/2 (jscs) (duration: 00m 12s)
* 21:36 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: temporarily revert T109756 (duration: 00m 11s)
* 21:02 csteipp: deployed patches for T108616 T91850 T91205 to wmf21 & 22
* 20:45 bblack: upgrading nginx to 1.9.4 on cp*
* 20:38 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 12s)
* 20:38 logmsgbot: ori@tin Synchronized php-1.26wmf22/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 15s)
* 20:37 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: wikimedia/cdb 1.2.0 → 1.3.0 (duration: 00m 14s)
* 20:07 logmsgbot: aude@tin Finished scap: Update group0 to new Wikidata branch (duration: 24m 27s)
* 19:42 logmsgbot: aude@tin Started scap: Update group0 to new Wikidata branch
* 19:14 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf21/: sync php-1.26wmf21 as well (duration: 02m 31s)
* 19:10 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf22
* 18:55 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
* 18:50 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.26wmf22 (duration: 29m 29s)
* 18:20 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.26wmf22
* 18:01 ejegg: rolled back payments to 6ac552f280fb839069d117386c4ecbe9e52f90a8
* 17:59 ejegg: updated payments from 6ac552f280fb839069d117386c4ecbe9e52f90a8 to 4c5e30288370db926cbbf7a7528edb9c41c65716
* 17:43 moritzm: enabled ferm on remaining hadoop workers (analytics1040-analytics1057)
* 17:09 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/NavigationTiming: T109756 (duration: 00m 11s)
* 16:56 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/CentralAuth: T108253 sul2 token store (duration: 00m 12s)
* 16:16 logmsgbot: ori@tin Synchronized php-1.26wmf21/vendor: I5af46eb3: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 14s)
* 15:43 logmsgbot: ori@tin Synchronized multiversion: wikimedia/cdb 1.0.1 → 1.2.0 (duration: 00m 12s)
* 15:21 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/236785/ (duration: 00m 12s)
* 15:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234910/ (duration: 00m 12s)
* 14:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1015 and es1019 (duration: 00m 11s)
* 14:30 moritzm: enabled ferm on hadoop workers up to analytics1039
* 12:41 godog: change whisper aggregation for 'sum.wsp' files T111170
* 10:48 moritzm: restarted salt master on palladium
* 10:32 logmsgbot: aude@tin Synchronized usagetracking.dblist: Enable usage tracking on Wikibooks (duration: 00m 11s)
* 09:55 moritzm: uploaded debdeploy 0.0.5 to carbon
* 04:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep  8 04:37:06 UTC 2015 (duration 37m 5s)
* 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-08 02:23:51+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 30s)
* 00:46 Krinkle: mwscript deleteEqualMessages.php --wiki eswiki
 
== 2015-09-07 ==
* 21:45 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/236682/ (duration: 00m 12s)
* 21:44 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/WikimediaEvents.php: https://gerrit.wikimedia.org/r/#/c/236196/1 (duration: 00m 12s)
* 21:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/WikiEditor: https://gerrit.wikimedia.org/r/#/c/236197/1 and https://gerrit.wikimedia.org/r/#/c/236679/ (duration: 00m 12s)
* 18:15 andrewbogott: graceful’d apache, restarted keystone on labcontrol1001
* 15:41 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: https://gerrit.wikimedia.org/r/#/c/236558/ (duration: 00m 12s)
* 15:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1004, pool es1018 (duration: 00m 10s)
* 10:04 godog: powercycle ms-be1003, loadavg skyrocketed
* 08:13 hashar: Jenkins upgraded to latest LTS ( https://phabricator.wikimedia.org/T111326 )
* 08:05 hashar: Upgrading Jenkins
* 04:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep  7 04:33:11 UTC 2015 (duration 33m 10s)
* 02:29 Krinkle: mwscript deleteEqualMessages.php --wiki pmswiki
* 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-07 02:23:27+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 22s)
 
== 2015-09-06 ==
* 04:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep  6 04:27:57 UTC 2015 (duration 27m 56s)
* 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-06 02:23:08+00:00
* 02:19 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 06m 14s)
 
== 2015-09-05 ==
* 23:37 Krinkle: mwscript deleteEqualMessages.php --wiki fywiktionary
* 04:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep  5 04:31:34 UTC 2015 (duration 31m 33s)
* 02:30 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-05 02:30:06+00:00
* 02:27 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 53s)
 
== 2015-09-04 ==
* 23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 12s)
* 23:52 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Beta-only change (duration: 00m 11s)
* 22:49 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/Citoid: https://gerrit.wikimedia.org/r/#/c/236218/ and https://gerrit.wikimedia.org/r/#/c/236222/ (duration: 00m 12s)
* 21:55 urandom: bouncing Cassandra on restbase1001 to restore default GC settings
* 18:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/236063/ (duration: 00m 11s)
* 18:06 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/extensions/WikimediaEvents/modules/ext.wikimediaEvents.statsd.js: Ib98988f67ef (duration: 00m 11s)
* 17:35 MaxSem: Maps: dropped duplicate index on water_polygons
* 16:27 jynus: cloning es1 mysql data from es1004 to es1018 [ETA:16h]
* 16:11 paravoid: updating firewall border ACLs and BGP border filters across all cr
* 15:42 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1002, es1016; Depool es1004 (duration: 00m 11s)
* 15:35 godog: python varnishlog collector + gdb running on cp1052 for debugging T83580
* 12:55 moritzm: restarted salt-master on palladium
* 12:47 moritzm: uploaded debdeploy 0.0.4 to carbon
* 10:18 logmsgbot: kartik@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/api/ApiContentTranslationPublish.php: php-1.26wmf21/extensions/ContentTranslation/extension.json T111490:Use the VirtualRESTService to configure CX (duration: 00m 12s)
* 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-fr-ca_1.0.3~r61329-1
* 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-fr_0.9.0~r28336-1
* 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-es_0.9.1~r60655-1
* 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-eo-ca_0.9.1~r60655-1
* 09:16 akosiaris: uploaded to apt.wikimedia.org trusty-wikimedia: apertium-ca-it_0.1.1~r57554-1
* 07:50 jynus: cloning es3 mysql data from es1008 to es1019
* 04:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep  4 04:19:20 UTC 2015 (duration 19m 19s)
* 02:26 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-04 02:26:04+00:00
* 02:23 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 21s)
* 01:56 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T111439 (duration: 00m 12s)
* 00:11 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/includes/resourceloader/ResourceLoader.php: I24f68e34a9fa4918 (duration: 00m 12s)
* 00:06 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235940/ (duration: 00m 11s)
 
== 2015-09-03 ==
* 23:53 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235853/ (duration: 00m 12s)
* 23:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
* 23:50 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/235843/ (duration: 00m 12s)
* 23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
* 23:40 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/ukwikivoyage.png: https://gerrit.wikimedia.org/r/#/c/235850/ (duration: 00m 12s)
* 23:37 mutante: mw1224 - killed and restarted defunct hhvm, version is different from the one on mw1225
* 23:37 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/235728 (duration: 00m 13s)
* 23:36 logmsgbot: krenair@tin Synchronized w/static/images/project-logos/knwikisource.png: https://gerrit.wikimedia.org/r/#/c/235728/ (duration: 00m 12s)
* 23:32 Krenair: mw1224 has been sending segfault warnings and "Lost parent, LightProcess exiting" to hhvm.log since about 21:17:34
* 23:29 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/235905/ (duration: 00m 13s)
* 23:28 logmsgbot: krenair@tin Synchronized php-1.26wmf21/package.json: bd2eb6cc1919c7dab056d5f8fe5b4a164236d78f (duration: 00m 13s)
* 23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235908/ (duration: 00m 13s)
* 21:21 ori: rebuilt HHVM with updated diff from facebook/hhvm PR #6071 (T109540), uploaded to apt as 3.6.5+dfsg1-1+wm5
* 21:18 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
* 19:54 bearND: MobileApps deployed sha1 553c399
* 19:31 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf21
* 18:13 ottomata: rolling restart of hadoop  yarn nodemanagers to pick up Yarn AppMaster port range limitation to apply ferm rules.
* 18:04 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
* 18:03 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Add plumbing code for Flow beta feature (unused for now) (duration: 00m 12s)
* 17:39 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235769/ (duration: 00m 12s)
* 17:34 mutante: bromine - deleting policy docroot
* 17:06 jynus: cloning es1006 mysql data into es1015 [ETA:8h]
* 16:30 bblack: updating nginx->1.9.4 on cp1071, cp3033 for prod validation before broader rollout
* 16:30 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es3 master switchover from es1009 to es1014 (eqiad) (duration: 00m 13s)
* 16:28 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es3 master switchover from es1009 to es1014 (codfw) (duration: 00m 13s)
* 16:26 mutante: imported jenkins 1.609.3 into APT repo
* 16:23 legoktm: fixed content model of Template:Languages@metawiki
* 16:21 robh: re-enabling puppet on all mw systems
* 16:14 robh: disabling puppet on all mw systems for apache config update
* 16:01 jynus: performing es3 master switchover from es1009 to es1014
* 15:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: depool es1006 (duration: 00m 12s)
* 15:17 hashar: stopping nodepool on labnodepool1001.eqiad.wmnet not ready yet
* 15:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: es2 master switchover from es1006 to es1011 (eqiad) (duration: 00m 13s)
* 15:14 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: es2 master switchover from es1006 to es1011 (codfw) (duration: 00m 12s)
* 15:05 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s)
* 15:04 logmsgbot: demon@tin Synchronized php-1.26wmf21/extensions/Translate/: (no message) (duration: 00m 15s)
* 14:51 jynus: performing es2 master switchover from es1006 to es1011
* 14:33 paravoid: rebooting msw1-eqiad
* 14:28 twentyafterfour: restarted phd (phabricator daemon) to pick up new configuration
* 14:25 paravoid: changing IPv6 RA interval/lifetime/virtual-router-only @ eqiad
* 14:21 paravoid: rebooting msw1-codfw
* 13:17 paravoid: upgrading mr1-esams and mr1-eqiad to newer junos
* 13:13 godog: bounce carbon daemons on graphite1001
* 12:42 chasemp: unban elastic1001 and put back in service
* 12:24 chasemp: move all shards off of elastic1001
* 12:24 chasemp: disable elastic1001 in lvs as we are gonig to try fw apply round #2
* 11:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1028; increase the load of es1010, es1013 and es1017 (duration: 00m 12s)
* 10:45 jynus: applying schema change for ContentTranslation on x1-master "wikishared"
* 10:02 godog: reenable puppet on ms-be1*
* 09:16 jynus: started profiling mysql queries at phabricator. Only a 1% overhead is expected.
* 09:12 moritzm: updated rsyncd firewall rules (see https://gerrit.wikimedia.org/r/235425 for details)
* 09:12 godog: stop puppet on ms-be1* after ferm rsync change
* 08:23 godog: fixup current graphite retention T96662
* 07:26 moritzm: enabled ferm on dbstore* servers in codfw
* 06:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep  3 06:29:35 UTC 2015 (duration 29m 34s)
* 03:09 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-03 03:09:20+00:00
* 03:06 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 32s)
* 02:45 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-03 02:45:36+00:00
* 02:39 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 41s)
* 01:32 logmsgbot: krenair@tin Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s)
* 00:36 logmsgbot: ori@tin Synchronized php-1.26wmf21/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 11s)
* 00:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 13s)
* 00:27 RoanKattouw: Deployed patch for T111029
 
== 2015-09-02 ==
* 23:58 logmsgbot: andyrussg@tin Synchronized php-1.26wmf20/extensions/CentralNotice/: CentralNotice update (duration: 00m 13s)
* 23:33 logmsgbot: andyrussg@tin Synchronized php-1.26wmf21/extensions/CentralNotice/: Update CentralNotice (duration: 00m 13s)
* 23:02 logmsgbot: andyrussg@tin Finished scap: Update CentralNotice to 2.6.0 for wmf21 (duration: 48m 18s)
* 22:13 logmsgbot: andyrussg@tin Started scap: Update CentralNotice to 2.6.0 for wmf21
* 20:27 arlolra: updated Parsoid to version 5f2fae6c
* 20:08 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.26wmf21
* 20:02 logmsgbot: krinkle@tin Synchronized php-1.26wmf21/resources/src/startup.js: Ie65427caee (duration: 00m 12s)
* 19:09 mutante: restarted gitblit, stopped counting
* 19:07 paravoid: upgrading mr1-codfw, mr1-ulsfo to newer junos
* 19:01 urandom: bouncing Cassandra on restbase1001 to address bogus icinga process failure alert
* 18:52 legoktm: deployed patch for T110553
* 18:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf21
* 18:32 cmjohnson1: replacing disk 10 on db1028
* 18:13 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
* 17:50 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/VisualEditor/modules/ve-mw/ui/inspectors: https://gerrit.wikimedia.org/r/#/c/235511/ (duration: 00m 12s)
* 17:07 logmsgbot: ori@tin Synchronized php-1.26wmf21/extensions/UniversalLanguageSelector: 78a5908fd9: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 16s)
* 17:07 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/UniversalLanguageSelector: 2154acc529: Updated mediawiki/core Project: mediawiki/extensions/UniversalLanguageSelector (duration: 00m 13s)
* 16:25 mutante: restarting NTP on lvs2004
* 16:12 jynus: setting BBU auto-learn mode to warn only (disabled if not possible) on all database hosts
* 16:03 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235484/ (duration: 00m 12s)
* 16:01 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235486/ (duration: 00m 12s)
* 15:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MultimediaViewer/MultimediaViewer.php: https://gerrit.wikimedia.org/r/#/c/235483/ (duration: 00m 13s)
* 15:56 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js: https://gerrit.wikimedia.org/r/#/c/235485/ (duration: 00m 12s)
* 15:51 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: T110837 (duration: 00m 13s)
* 15:42 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235482/ (duration: 00m 12s)
* 15:34 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/OpenStackManager/nova/OpenStackNovaController.php: https://gerrit.wikimedia.org/r/#/c/235479/ (duration: 00m 13s)
* 15:19 logmsgbot: krenair@tin Synchronized php-1.26wmf21/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235442/ (duration: 00m 12s)
* 15:14 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/ContentTranslation/modules/tools/ext.cx.tools.template.js: https://gerrit.wikimedia.org/r/#/c/235441/ (duration: 00m 12s)
* 15:07 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234942/ and https://gerrit.wikimedia.org/r/#/c/234944/ (duration: 00m 13s)
* 14:40 Nikerabbit: TTMServer reindex complete
* 11:59 mark: removed tools LV snapshots on labstore1002
* 11:47 mark: kill STOP'ed rsync on labstore1002
* 11:00 jynus: cloning mysql data from es1002 into es1016 [ETA:16h]
* 10:30 moritzm: installed qemu security updates on labvirt*
* 09:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1002 (duration: 00m 12s)
* 09:21 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1010, pool es1017 (duration: 00m 13s)
* 09:19 hashar: Merged in "delete 1.26wmf12" https://gerrit.wikimedia.org/r/235347 which was left unmerged in Gerrit but was present on tin /srv/mediawiki-staging confusing people.
* 08:03 bblack: restarting ntp on lvs2004
* 08:01 moritzm: enable ferm on db1069/sanitarium
* 07:50 moritzm: enable ferm on remaining phabricator db hosts
* 04:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep  2 04:54:37 UTC 2015 (duration 54m 36s)
* 02:52 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-02 02:52:51+00:00
* 02:50 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 09s)
* 02:29 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-02 02:29:56+00:00
* 02:26 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 31s)
* 00:33 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235366/ (duration: 00m 13s)
 
== 2015-09-01 ==
* 23:59 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/221731/ (duration: 00m 13s)
* 23:41 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235285/ (duration: 00m 14s)
* 23:08 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/235362/ (duration: 00m 14s)
* 23:02 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/235361/ (duration: 00m 13s)
* 22:50 awight: update CRM from 0fc8474338e7a31fdde79287bd667b98cd96a252 to abc34b87ee9d1dbb1176f1929a3d748e1ee5ac7b
* 22:18 MaxSem: Maps: creating and populating admin table
* 21:20 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/235177/ (duration: 00m 12s)
* 20:54 ori: restarted nutcracker on mw1142
* 20:33 logmsgbot: twentyafterfour@tin Finished scap: sync 1.26wmf21 (duration: 30m 37s)
* 20:03 dcausse: unfreezing elasticsearch indices
* 20:03 logmsgbot: twentyafterfour@tin Started scap: sync 1.26wmf21
* 19:52 YuviPanda: removed tools20150901132642 from labstore vg on labstore1002
* 19:36 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/skins/SkinTemplate.php: cc643a0934: Deprecate unconditional loading of mediawiki.ui.button on all pages (duration: 00m 13s)
* 17:31 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
* 17:28 dcausse: freezing elasticsearch indices before applying ferm fules on master
* 17:23 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata: Fix for change dispatcher (duration: 00m 20s)
* 16:45 jynus: performing schema change on testwiki and metawiki
* 16:12 robh: policy.wikimedia.org dns change happening now
* 16:00 chasemp: ferm for elastic1003/2/1(master)
* 15:57 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235168/ (duration: 00m 13s)
* 15:51 YuviPanda: stopped replicate-tools on labstore1002, and cleaned out lockdir
* 15:47 logmsgbot: reedy@tin Synchronized php-1.26wmf20/extensions/SecurePoll/: Stop cronspam (duration: 00m 13s)
* 15:47 mark: labstore1002: echo 10000 > /sys/block/md123/md/sync_speed_min
* 15:44 mark: labstore1002: update-initramfs -k all -u
* 15:38 mark: labstore1002: mdadm /dev/md/slice51 --add /dev/sd{bh,bg,bf,be,bd,bc}
* 15:36 moritzm: disabled ferm in analytic1028, needs some more work on possibly dynamic mapreduce ports
* 15:16 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sd{bb,ba,az}
* 15:14 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sdaw
* 15:07 mark: labstore1002: mdadm --zero-superblock /dev/sd{aw,bh,bg,bf,be,bd,bc,bb,ba,az}1
* 15:04 moritzm: enabled ferm in analytic1028 (initial hadoop worker)
* 15:04 mark: labstore1002: mdadm --zero-superblock /dev/sdax1 && mdadm /dev/md/slice15 --re-add /dev/sdax
* 15:03 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231465/ - VE for all new enwiki accounts (duration: 00m 13s)
* 14:58 mark: labstore1002: mdadm /dev/md/slice15 --re-add /dev/sday
* 14:58 mark: labstore1002: mdadm --zero-superblock /dev/sday1
* 14:53 mark: labstore1002: mdadm --stop /dev/md3
* 14:37 ebernhardson: reset elasticsearch cluster.routing.allocation.disk.high back to 90%
* 13:38 logmsgbot: krinkle@tin Synchronized w/: Remove rl-test.php (duration: 00m 13s)
* 13:17 moritzm: enabled ferm on db1048
* 13:09 moritzm: enabled ferm on labsdb100[467]
* 12:01 YuviPanda: disable puppet on labsdb1006
* 08:58 moritzm: enabled ferm on labsdb1001
* 08:58 godog: fixup current graphite retention for metrics under "servers" hierarchy T96662
* 08:51 moritzm: enabled ferm on labsdb1002
* 08:31 moritzm: enabled ferm on labsdb1003
* 08:29 godog: repool mw1125 mw1142 after nutcracker failures
* 07:45 jynus: cloning mysql data from es1010 to es1017 [ETA: 6h]
* 07:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1010 (duration: 00m 12s)
* 07:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool es1007, pool es1013 (duration: 00m 13s)
* 06:36 mutante: uploaded survey2012 to dumps/dataset1001; ownership as it is for survey2011; - T110746 in time for midnight PST
* 05:18 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep  1 05:18:09 UTC 2015 (duration 18m 8s)
* 02:28 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-09-01 02:28:30+00:00
* 02:25 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 00s)
 
== 2015-08-31 ==
* 23:56 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233665/ (duration: 00m 11s)
* 23:49 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: reenable config changes for cirrus experimental completion api (duration: 00m 12s)
* 23:40 logmsgbot: ori@tin Synchronized php-1.26wmf20/extensions/EducationProgram: 97ab82eab2: Updated mediawiki/core Project: mediawiki/extensions/EducationProgram  85a7d3932c1a4ad28f1a8dd05704f4e524152349 (duration: 00m 14s)
* 23:27 logmsgbot: ebernhardson@tin Synchronized php-1.26wmf20/extensions/CirrusSearch/: (no message) (duration: 00m 12s)
* 23:25 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: revert update for cirrussearch experimental suggestions api (duration: 00m 12s)
* 23:21 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: update config of cirrussearch experimental suggestions api (duration: 00m 12s)
* 22:45 chasemp: disabled puppet on elastic hosts temporarily to safely roll out fw change.  elastic seems to have not taken it well and I'm holding for green cluster state.
* 21:20 mutante: installing package upgrades on argon
* 20:58 ori: imported pybal_1.08_amd64.changes to jessie-wikimedia
* 20:44 chasemp: ferm for elastic100[4-7] and adjust ferm to include wikitech source
* 20:21 subbu: deployed parsoid version c3e4df5e
* 16:22 godog: depool mw1125 + mw1142 from api, nutcracker client connections exceeded
* 16:06 logmsgbot: thcipriani@tin Finished scap: SWAT: Ask the user to log in if the session is lost [[gerrit:234228]] (duration: 27m 07s)
* 15:59 jynus: restarting hhvm on mw2187
* 15:39 logmsgbot: thcipriani@tin Started scap: SWAT: Ask the user to log in if the session is lost [[gerrit:234228]]
* 15:33 mutante: terbium - Could not find dependent Service[nscd] for File[/etc/ldap/ldap.conf]
* 15:28 logmsgbot: thcipriani@tin Synchronized closed-labs.dblist: SWAT: Creating closed-labs.dblist and closing es.wikipedia.beta.wmflabs.org [[gerrit:234594]] (duration: 00m 13s)
* 15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Remove files from Commons from search results on wikimediafoundation.org [[gerrit:234040]] (duration: 00m 11s)
* 15:25 ottomata: starting varnishkafka instances on frontend caches to produce eventlogging client side events to kafka
* 15:21 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - Fix formatting of client edit summaries [[gerrit:234991]] (duration: 00m 21s)
* 15:16 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/UploadWizard/resources/controller/uw.controller.Step.js: SWAT: Keep the uploads sorted in the order they were created in initially [[gerrit:234553]] (duration: 00m 12s)
* 14:43 ebernhardson: elasticsearch cluster.routing.allocation.disk.watermark.high set to 75% to force elastic1022 to reduce its disk usage
* 14:41 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
* 14:06 akosiaris: rebooted krypton. was reporting 100% cpu steal time
* 13:40 paravoid: running puppet on newly-installed mc2001
* 13:40 paravoid: restarting hhvm on mw1065
* 11:10 moritzm: restart salt-master on palladium
* 10:45 paravoid: reenabling asw2-a5-eqiad:xe-0/0/36 (T107635)
* 10:36 godog: repool ms-fe1004
* 10:32 godog: repool ms-fe1003 and depool ms-fe1004 for firewall changes
* 10:19 godog: update graphite retention policy on files with previous retention and older than 30d T96662
* 10:18 godog: repool ms-fe1002 and depool ms-fe1003 for firewall changes
* 10:05 godog: depool ms-fe1002 to apply firewall changes
* 09:55 jynus: cloning es1007 mysql data into es1013 (ETA: 5h30m)
* 09:51 godog: repool ms-fe1001
* 09:35 godog: depool ms-fe1001 in preparation for ferm changes
* 09:27 godog: update graphite retention policy on files with previous retention and older than 60d T96662
* 09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1007 for maintenance (duration: 00m 13s)
* 08:33 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 12s)
* 04:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 31 04:34:14 UTC 2015 (duration 34m 13s)
* 04:05 bblack: disabled ipv6 autoconf on neon, flushed old dynamic addr
* 02:32 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-31 02:32:25+00:00
* 02:29 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 42s)
 
== 2015-08-30 ==
* 12:58 godog: lvchange -ay labstore/others on labstore1002
* 12:52 godog: start-nfs on labstore1002
* 12:31 godog: lvchange -ay labstore/tools on labstore1002
* 12:30 godog: also disabled puppet on labstore1002 while investigating
* 12:15 godog: trying to manually assemble missing raid on labstore1002 with mdadm --assemble /dev/md/slice51 --uuid 0747643d:b89b36ff:57156095:c33694fc --verbose
* 11:19 YuviPanda: powered labstore1002 back up
* 11:17 YuviPanda: shut down labstore1002, going to powercycle from mgmt
* 10:34 YuviPanda: disabled backups on labstore1002 to prevent overwriting of good backups on 2001
* 10:08 YuviPanda: rebooted labstore1002
* 04:16 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 30 04:16:17 UTC 2015 (duration 16m 16s)
* 02:23 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-30 02:23:07+00:00
* 02:20 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 36s)
 
== 2015-08-29 ==
* 15:26 jynus: killing idle mysql connections from phabricator and setting wait and interactive timeout to 60
* 09:30 jynus: SCAP failed, cannot depool db1028
* 09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
* 09:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s)
* 09:05 jynus: about to depool db1028 due to disk issue
* 04:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s)
* 02:24 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00
* 02:21 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s)
 
== 2015-08-28 ==
* 23:45 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234679/ (duration: 06m 56s)
* 22:51 logmsgbot: bd808@tin Synchronized wmf-config/CommonSettings-labs.php: Use ffmpeg instead of avconv on labs beta (I250fe33) (duration: 06m 05s)
* 22:05 ori: disabling puppet on tin for a few minutes to test an ssh-agent-proxy change
* 20:04 logmsgbot: catrope@tin Synchronized php-1.26wmf20/resources/src/mediawiki.legacy/shared.css: T110716 (duration: 00m 12s)
* 18:09 robh: updating ldap-codfw cert
* 17:10 logmsgbot: catrope@tin Synchronized php-1.26wmf20/extensions/Flow/includes/Parsoid/Utils.php: T110676 (duration: 00m 13s)
* 17:08 urandom: bouncing Cassandra on restbase1001 to apply default (puppet-managed) settings
* 16:03 chasemp: ferm for elasticsearch10(0[8-9|1[0-13])
* 15:31 awight: updated crm from fc0fcc8f5af262b56392d3f4f5998f8ea08c99a8 to 0fc8474338e7a31fdde79287bd667b98cd96a252
* 15:23 chasemp: ferm for elasticsearch10[14-17]
* 11:09 logmsgbot: aude@tin Synchronized php-1.26wmf20/extensions/Wikidata/Wikidata.php: Sync entry point - updated to work on Jenkins together with ContentTranslation (duration: 00m 12s)
* 10:29 godog: reenable puppet on ms-fe1, ferm changes will go out on monday
* 09:48 jynus: Cloning es1001 database into es1012
* 09:45 moritzm: enabled ferm for swift on esams
* 09:28 moritzm: enabled ferm on strontium puppetmaster backend
* 09:00 moritzm: enabled ferm on rhodium puppetmaster backend
* 08:29 moritzm: uploaded debdeploy 0.0.3 to carbon
* 08:23 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1001, increas weight of es1011, pool es1014 for the first time (duration: 00m 13s)
* 05:59 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 28 05:59:09 UTC 2015 (duration 59m 8s)
* 04:58 logmsgbot: ori@tin Synchronized php-1.26wmf20/includes/parser/Parser.php: 754b222daf: Add ParserOutput cache and expiry times to NewPP report (duration: 00m 13s)
* 02:41 logmsgbot: l10nupdate@tin LocalisationUpdate completed (1.26wmf20) at 2015-08-28 02:41:26+00:00
* 02:35 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 47s)
* 01:59 Tim: on ruthenium: started parsoid_vd which was previously killed by oom-killer
* 01:58 Tim: on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things
* 01:37 Tim: on ruthenium restarted parsoid-rt-client and parsoid-vd-client
* 00:24 mutante: powercycled mw2027
* 00:19 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s)
* 00:06 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s)
* 00:05 Krenair: Another codfw host broke: mw2027
* 00:01 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234330/ (duration: 00m 13s)
 
== 2015-08-27 ==
* 23:58 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/MobileFormatter.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 12s)
* 23:57 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/config/Experimental.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 14s)
* 23:55 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233439/ (duration: 00m 12s)
* 23:30 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Gadgets/extension.json: touch (duration: 00m 13s)
* 23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/DefaultSettings.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
* 23:24 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/registration/ExtensionProcessor.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s)
* 23:23 logmsgbot: krenair@tin Synchronized php-1.26wmf20/includes/MWNamespace.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 13s)
* 23:15 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/234009/ (duration: 00m 13s)
* 23:04 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233100/ (duration: 00m 12s)
* 20:11 chasemp: ferm setup on elasticsearch10(1[8-9|2[0-3])
* 20:06 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf20
* 19:57 logmsgbot: twentyafterfour@tin Synchronized php-1.26wmf20/includes/media/XMP.php: deploy fix for T89532 on 1.26wmf20 (duration: 00m 13s)
* 18:16 chasemp: setting up ferm on elastic1027-31
* 17:47 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 13s)
* 17:43 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234320/2 (duration: 00m 13s)
* 17:37 urandom: ack'd Cassandra process alert on restbase1001; temporary command args have pushed the class name beyond the limit
* 17:34 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: (no message) (duration: 00m 12s)
* 17:24 logmsgbot: krenair@tin Synchronized multiversion/MWMultiVersion.php: https://gerrit.wikimedia.org/r/#/c/234320/ (duration: 00m 12s)
* 17:08 urandom: bouncing Cassandra on restbase1001 to apply temporary GC settings
* 16:51 moritzm: ferm rules on logstash100[1-3] have been amended to allow grafana from reading dashboard configs
* 16:39 bd808: new ferm rules on logstash100[1-3] are blocking grafana from reading dashboard configs.
* 16:22 moritzm: ferm enabled on logstash1003
* 16:18 moritzm: ferm enabled on logstash1002
* 16:16 bd808: ferm enabled on logstash1001
* 16:06 bd808: logstash1001 back up after system reboot; we applied a default drop rule without applying the other iptables changes; will try again
* 15:58 chasemp: rebooting logstash1001.mgmt.eqiad.wmnet for moritz as it is having issues
* 15:47 bblack: killed hung ubuntu mirror rsync commands on carbon, from Jul 10
* 15:45 bd808: logstash1001 not responding over ssh following ferm rules application; moritzm investigating
* 15:30 bd808: Disabled puppet on logstash100[1-3] prior to trying to enable ferm
* 15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable newarticle campaign in itwiki [[gerrit:234223]] (duration: 01m 52s)
* 14:52 bblack: re-imaging lvs200[123]
* 14:47 godog: reenable puppet on ms-be1*
* 14:22 godog: disable puppet on ms-fe1 / ms-be1 in prepration for puppet work
* 14:15 godog: reenable puppet on ms-fe2*
* 13:47 bblack: re-imaging lvs2004 + lvs2005
* 13:29 ottomata: doing rolling restart of kafka brokers to apply auto_create_topics change
* 13:21 godog: enable puppet on ms-be2*
* 13:21 ottomata: stopping kafka on analytics1021, it is no longer a kafka broker.
* 13:09 godog: disable puppet on ms-be2* in preparation for firewall changes
* 13:09 jynus: cloning es1008 into es1014
* 13:04 ottomata: running leader election now that all topics and partitions are rebalanced across new kafka nodes
* 12:46 bblack: re-imaging lvs2006
* 12:45 andrewbogott: re-imaging labnet1001 (I hope)
* 11:33 _joe_: restarted hhvm on mw1143, locked in __lll_lock_wait for stat_cache deadlock
* 11:10 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool es1011 for the first time, depool es1008 (duration: 00m 12s)
* 09:27 jynus: installing and configuring servers es1012-es1019
* 06:39 ostriches: tin: dropped useless "gerrit" remote from /srv/mediawiki-staging (uses ssh, lol), pointed {origin,readonly} at the actual repo instead of a redirect.
* 06:00 _joe_: powercycling mw2140, not responding to ping, blank console
* 03:17 awight: deploy config cleanup for paymentswiki
* 02:38 logmsgbot: l10nupdate@tin Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 10m 44s)
* 02:16 awight: push config change to the payments orphan slayer: explitly give stomp port to work around strict notice, clean up unused globals. T109911
* 01:32 ejegg: updated payments from 8ba4b5299f195cf48e6809b18a21e2d53f6eec1b to 6ac552f280fb839069d117386c4ecbe9e52f90a8
* 00:31 twentyafterfour: finished phabricator upgrade, everything appears to be working
* 00:24 logmsgbot: aaron@tin Synchronized php-1.26wmf19/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
* 00:22 logmsgbot: aaron@tin Synchronized php-1.26wmf20/extensions/CentralAuth: 47e181adb2898977b146de7398eaa35aebb870e3 (duration: 01m 13s)
* 00:20 twentyafterfour: taking phabricator offline for scheduled upgrade
 
== 2015-08-26 ==
* 23:59 Krinkle: mwscript deleteEqualMessages.php --wiki rowiki
* 23:57 yurik: git deployed tilerator - had the 4/5 issue - https://phabricator.wikimedia.org/T110434
* 23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234072/ (duration: 01m 12s)
* 23:37 logmsgbot: krenair@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234038/ (duration: 01m 12s)
* 23:35 logmsgbot: krenair@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: https://gerrit.wikimedia.org/r/#/c/234037/1 (duration: 01m 12s)
* 23:27 yurik: deployed kartotherian
* 23:21 jynus: cloning es1005 into es1011, ETA 9 hours
* 22:41 ori: armed keyholder on tin
* 22:40 ori: Disabled Puppet on mw1017 for 2hrs and applied I059b0c96c9 for testing.
* 21:55 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
* 21:48 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool es1005 (duration: 01m 12s)
* 21:40 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 01m 12s)
* 21:32 ori: Disabling Puppet on tin again to test an ssh-agent-proxy change
* 20:30 logmsgbot: ori@tin Synchronized README: testing ssh-agent-proxy changes (duration: 00m 13s)
* 20:25 ori: Disabling puppet on tin and hacking some debug logging into ssh-agent-proxy
* 20:24 ori: armed ssh-agent key on mira
* 20:21 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/includes/poolcounter/PoolWorkArticleView.php: (no message) (duration: 00m 03s)
* 20:11 subbu: deployed parsoid version 44d657de
* 19:52 logmsgbot: krenair@tin Synchronized php-1.26wmf20/extensions/Echo/includes/mapper/EventMapper.php: https://gerrit.wikimedia.org/r/#/c/234082/ (duration: 00m 12s)
* 19:47 mutante: sodium - deleting shunted messages older than 7 days
* 19:23 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234042/ (duration: 00m 12s)
* 19:22 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/234024/ (duration: 00m 12s)
* 19:20 logmsgbot: krenair@tin Synchronized multiversion/MWWikiversions.php: https://gerrit.wikimedia.org/r/#/c/232672/ (duration: 00m 12s)
* 18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf20/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 11s)
* 18:50 logmsgbot: krinkle@tin Synchronized php-1.26wmf19/maintenance/deleteEqualMessages.php: (no message) (duration: 00m 13s)
* 18:38 twentyafterfour: ^ stupid typo.  That sync was group1 to 1.26wmf20
* 18:37 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: tig
* 18:31 logmsgbot: ori@tin Synchronized w/404.php: Ided1facc0: Remove auto-redirection from 404 page. (duration: 00m 13s)
* 17:51 ejegg: updated SmashPig from 258f2c917b1ae50b01231927bcd6f58ecaa8940b to fdb053efa617162ac9f695e493c390987a069140
* 17:30 urandom: bouncing Cassandra on restbase1001 to apply temporary GC setting
* 17:12 andrewbogott: ok, /now/ I’m running a dist-upgrade on labcontrol1001, to sort out weird oslo dependencies
* 17:09 chasemp: adding firewall to elasticsearch2[4-6] (3 was just done as a pilot)
* 17:03 andrewbogott: upgraded labnet1002 nova services to Juno
* 16:34 andrewbogott: stopping keystone, updating db, restarting
* 16:18 andrewbogott: switching labcontrol1001 hiera to Juno which will add the cloud-archive repo for Juno.
* 16:11 andrewbogott: backing up labs openstack databases into /home/andrew/openstackdbbackups on db1009
* 16:11 andrewbogott: starting labs openstack update to Juno
* 15:53 moritzm: ferm enabled on elastic1023
* 15:45 godog: repool restbase1009 in pybal
* 15:28 logmsgbot: thcipriani@tin Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - wrap usage tracking batch updates in transaction [[gerrit:233970]] (duration: 00m 23s)
* 13:47 andrewbogott: rebooting/reimaging labnet1001
* 13:11 mobrovac: restbase deploying 1dfba85
* 12:54 yurik: git synced kartotherian
* 11:02 jynus: dropping optin_survey_old table on all wikis
* 10:33 godog: reenable puppet on ms-fe/ms-be, base::firewall still not enabled
* 09:58 godog: test-reboot ms-be2001
* 08:17 godog: disable puppet on ms-be/ms-fe in preparation for merging firewall changes